Jacqueline Kory-Westlund

Click here to see the video showing this project!

Study Overview

For my master's thesis at the MIT Media Lab, I created a social robotic learning companion that played a storytelling game with young kids.

Children’s oral language skills in preschool can predict their academic success later in life. Helping children improve their language and vocabulary skills early on could help them succeed later. Furthermore, language learning is a highly social, interactive activity. When creating technology to support children's language learning, technology that leverages the same social cues and social presence that people do—such as a social robot—will likely provide more benefit than using technology that ignores the critical social aspects of language learning.

As such, in this project, I examined the potential of a social robotic learning companion to support children's early long-term language development.

Study

The robot was designed as a social character, engaging children as a peer, not as a teacher, within a relational, dialogic context. The robot targeted the social, interactive nature of language learning through a storytelling game that the robot and child played together. The game was on a tablet—the tablet showed a couple characters that the robot or child could move around while telling their story, much like digital stick puppets. During the game, the robot introduced new vocabulary words and modeled good story narration skills.

Furthermore, because children may learn better when appropriately challenged, we asked whether a robot that Matched the “level” of complexity of the language it used to the general language ability of the child might help children improve more. For half the children, the robot told easier or harder stories based on an assessment of the child’s general language ability.

17 preschool children played the storytelling game with the robot eight times each over a two-month period.

I evaluated children's perceptions of the robot and the game, as well as whether the robot's matching influenced (i) whether children learned new words from the robot, (ii) the complexity and style of stories children told, and (iii) the similarity of children’s stories to the robot’s stories. I expected that children would learn more from a robot that matched, and that they would copy its stories and narration style more than they would with a robot that did not match. Children’s language use was tracked across sessions.

Results

I found that all children learned new vocabulary words, created new stories during the game, and enjoyed playing with the robot. In addition, children in the Matched condition maintained or increased the amount and diversity of the language they used during interactions with the robot more than children who played with the Unmatched robot.

Understanding how the robot influences children’s language, and how a robot could support language development will inform the design of future learning/teaching companions that engage children as peers in educational play.

Links

• This work is also on the Personal Robots Group site and the MIT Media Lab site
• New Scientist covered this work: Kindergarten bots teach language to tots
• I wrote about this project for the MIT Media Lab blog: Making new (robot) friends: Understanding children's relationships with social robots.
• That blog post was republished on IEEE Spectrum: Robots for Kids: Designing Social Machines That Support Children's Learning.

Publications

• Kory, J. (2014). Storytelling with robots: Effects of robot language level on children's language learning. Master's Thesis, Media Arts and Sciences, Massachusetts Institute of Technology, Cambridge, MA. [PDF]

• Kory, J., & Breazeal, C. (2014). Storytelling with Robots: Learning Companions for Preschool Children’s Language Development. In P. A. Vargas & R. Aylett (Eds.), Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE: Washington, DC. [PDF]

• Kory-Westlund, J., & Breazeal, C. (2015). The Interplay of Robot Language Level with Children's Language Learning during Storytelling. In J. A. Adams, W. Smart, B. Mutlu, & L. Takayama (Eds.), Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction: Extended Abstracts (pp. 65-66). [on ACM]

• Kory-Westlund, J. (2015). Telling Stories with Green the DragonBot: A Showcase of Children's Interactions Over Two Months. In J. A. Adams, W. Smart, B. Mutlu, & L. Takayama (Eds.), Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction: Extended Abstracts (p. 263). [on ACM] [PDF] [Video] Winner of Best Video Award.

• Kory-Westlund, J. M., & Breazeal, C. (2019). Exploring the effects of a social robot's speech entrainment and backstory on young children's emotion, rapport, relationships, and learning. Frontiers in Robotics and AI, 6. [PDF] [online]

Summer at NASA

In 2011, the summer after I graduated college, I headed to Greenbelt, Maryland to work with an international team of engineers and computer scientists at NASA Goddard Space Flight Center. The catch: we were all students! Over forty interns from at least four countries participated in Mike Comberiate's Engineering Boot Camp.

• Project overview
• Articles and other media
• Videos

Overview

The boot camp included several different projects. The most famous was GROVER, the Greenland Rover, a large autonomous vehicle that's now driving across the Greenland ice sheets mapping and exploring.

The main project I worked on was called LARGE: LIDAR-Assisted Robotic Group Exploration. A small fleet of robots -- a mothership and some workerbots -- used 3D LIDAR data to explore novel areas. My software team developed object recognition, mapping, path planning, and other software autonomously control the workerbots between infrequent contacts with human monitors. We wrote control programs using ROS.

Later in the summer, we presented demonstrations of our work at both NASA Wallops Flight Facility and at NASA Goddard Space Flight Center.

The LARGE team

• Mentors: NASA Mike, Jaime Cervantes, Cornelia Fermuller, Marco Figueiredo, Pat Stakem

• Software team: Felipe Farias, Bruno Fernades, Thomaz Gaio, Jacqueline Kory, Christopher Lin, Austin Myers, Richard Pang, Robert Taylor, Gabriel Trisca

• Hardware team: Andrew Gravunder, David Rochell, Gustavo Salazar, Matias Soto, Gabriel Sffair

• Others involved: Mike Huang, William Martin, Randy Westlund

Project description

The goal of the LARGE project is to assemble a networked team of autonomous robots to be used for three-dimensional terrain mapping, high-resolution imaging, and sample collection in unexplored territories. The software we develop in this proof-of-concept project will be transportable from our test vehicles to actual flight vehicles, which could be sent anywhere from toxic waste dumps or disaster zones on Earth to asteroids, moons, and planetary surfaces beyond.

The robot fleet consists of a single motherbot and a set of workerbots. The motherbot is capable of recognizing the location and orientation of each workerbot, allowing her to designate target destinations for any worker and track their progress. Presently, localization and recognition is performed via the detection of spheres mounted in a unique configuration atop each robot. Each worker can independently plot a safe path through the terrain to the goal assigned by the motherbot. Communication between robots is interdependent and redundant, with messages sent over a local network. If communication between workers and the motherbot is lost, the workers will be able to establish a new motherbot and continue the mission. The failure of any single robot or device will not prevent the mission from being completed.

The robots use LIDAR sensors to take images of the terrain, stitching successive images together to create global maps. These maps can then be used for navigation. Eventually, several of the workers will carry other imaging sensors, such as cameras for stereo vision or a Microsoft Kinect, to complement the LIDAR and enable the corroboration of data across sensory modalities.

Articles and other media

In the media:

• Geeked on Goddard reported on the lab all summer! A series of five articles appears here. With videos!

• GROVER, the Greenland Rover that was being developed by some of the students in the boot camp, headed out to the ice sheet in May 2013!

On my blog:

• Don't ever stop - the start of the summer, an overview of the projects, and an inspiring quotes

• Computer innards and radios - in which I learn many new things

• Science vs. engineering - observations on myself vs. the engineers I worked with

• Trip to Wallops - the lab took a trip to the beach to test out GROVER

• Engineering Boot Camp videos - it seems some of these videos no longer work, but see the others below!

Videos

I spent the summer writing code, learning ROS, and dealing with our LIDAR images. Other people took videos! (Captions, links to videos, & credits are below the corresponding videos.) More may be available on Geeked on Goddard or from nasagogblog's youtube channel.

• Meet the robots: A quick slideshow of the robots in the lab. Video from nasagogblog.

• Getting ready for the beach: Preparations for our trip to NASA Wallops, where GROVER was tested on the beach. Video from nasagogblog.

• GROVER on the beach: The Greenland ROVER during a test run on the beach, during our trip to Wallops. Video from nasagogblog.

Questions vs answers

Recently, I had a discussion with a friend about the key difference between science and engineering.

As a computer engineer, my friend found that the more advanced his coursework got and the more he learned about electronics, circuits, and microprocessors, the better he understood the subjects as a whole.

Which shouldn't be too surprising. That's the point of a college engineering degree: learn how stuff works and how to make stuff work.

But me, I find that as I learn more about brains and minds, filled with complex interactions between neurons, glial cells, neurotransmitters, and hormones, the picture gets steadily more complicated. The universe is one big dynamic system, full of chaotic pieces, and I keep finding more questions. The more I learn, the less I know.

That's the scientist's perspective on the world: more knowledge means more questions. More astonishment, more confusion.

(This is not a novel pronouncement, merely a recent observation supporting previously suggested differences between the two disciplines.)

A month after graduation, I'm well on my way to learning all sorts of crazy new things. This summer, I'm learning about...

• HAM radio. On Tuesday, I attended the first of a summer-long amateur radio FCC licensing class. I know very little about radios and their components - the president of GSFC's amateur radio club told a story about how easy it was to build a circuit to convert 5 volts down to 3.3 volts, and kept throwing out electronics jargon. I'm looking forward to increasing my knowledge of the subject!

• Computer innards. On a similarly technical note, my laptop's hard drive stopped spinning up last week. With the help of a computer engineering friend, I opened up the laptop and replaced the drive. Didn't even lose a screw! It's a small step into the world of computer hardware, but that was the first time I've opened up a computer, so it counts for a lot.

• Multiple realizability. That is, that people can take entirely different paths to the same place. People with ridiculously different beliefs can still be thinking exactly the same thing at exactly the same time on ridiculously frequent occasions.

• Tae Kwon Do. An activity I'd never done before: martial arts! All the interns/apprentices in my lab this summer were encouraged to try it out, since the GSFC club is so friendly. We've learned miscellaneous self-defense maneuvers and more ways of kicking than I remember names for - I even got to kick through a board!

• And software... My lab group is using a variety of software tools and open source code libraries that are new to me: ROS (the Robot Operating System), a code repository via SVN, the MRPT libraries, the point cloud library (PCL), and many more. I'm remembering C++, delving into path planning algorithms, and reading up on SLAM (simultaneous localization and mapping). Yes, it's a whirlwind of acronyms.

Don't ever stop

This one's a life update post, but it's also a "here's some cool science!" post.

A few days ago, I graduated from Vassar College with a Bachelor of Arts in Cognitive Science and a correlate in Computer Science. I was decorated with general honors, departmental honors, membership to Psi Chi, and membership to Sigma Xi. My time there was awesome.

What's next?

No lazy summer!

Well, no lazy summer break for me! I've already spent three days in my summer lab at NASA Goddard Space Flight Center, where I'll be working on a number of software development projects. The primary one is a LIDAR-assisted robotic group exploration project, in which we're going to have a small fleet of robots -- a mothership and some workerbots -- use 3D LIDAR data to autonomously map and plot paths through an area. This kind of robot fleet could, eventually, be used to explore other planets. One of the big challenges will be dealing with the 3D image data. I'm looking forward to learning more image processing algorithms!

Another project is the redesign of the Greenland Robotic Vehicle, a big autonomous rover that'll drive across Greenland, collecting a data about snowfall, mapping, and exploring. Did you know there's ice on that country two miles thick? I may also get to play with a robot that has stereo vision.

You can see some of these robots (and what life in the lab may be like) in this great video about last year's interns.

So far, I've met a bunch of intelligent, friendly folks, started catching up on already-written code, and begun to delve into the platforms, libraries, and algorithms we'll be using and developing this summer. Our mentors have already proven themselves to be enthusiastic and helpful. Just yesterday, one of them told us,

"You're engineers at NASA. You want to go where things are, and then go beyond."

That may end up being our theme for the summer.

A little overwhelming?

There's going to be so much going on. It'd be easy to get overwhelmed -- especially now, jumping in and floundering around in the code, the projects, the people. So much to learn.

But as I sat in the lab today, reading about ROS, going through tutorials, reading about PCL and feature detection in point clouds, digging through last summer's confusing pile of C# and C++ programs, I realized I wasn't overwhelmed. And it was because of all the other experiences I've had that've gotten me to this point.

Confidence. My first URSI summer, flailing through Microsoft Robotics Studio and complicated conceptual theories. Figuring out how to deal with webcams and image data my second URSI summer, reading papers on optical flow and implementing algorithms. Last summer: excavations of an open source flight simulator, the Aeronautics Student Forum, dealing with different work styles and communication styles in my LARSS lab. And more.

I think about all those experiences, and I'm not afraid of this summer. I could almost be overwhelmed -- perhaps thinking that everyone else has more of the right kind of experience; I wasn't trained as a classic engineer -- but I know I can succeed. My non-engineering, cognitive science background sets me apart and lets me look at problems a little differently than everyone else. I'm an asset.

I know how to learn. I know how to do research.

I can conquer this summer.