December 11, 2019

Virtual Actor Pattern in Go

How do you build an actor in such a way that it's considered virtual?

The actor model in computer science is a mathematical model of concurrent computation that treats "actor" as the universal primitive of concurrent computation. In response to a message it receives, an actor can: make local decisions, create more actors, send more messages, and determine how to respond to the next message received. Actors may modify their own private state, but can only affect each other indirectly through messaging (obviating lock-based synchronization).

The actor pattern is a well-established pattern within computing. The premise of an actor is the ability to pass a message and act on it independently of the original sender. The value of the actor pattern is that you can decouple an implementation from the data, leaving you with a data-oriented design pattern. Data-oriented design helps focus an application around the data it's operating on, and is generally a good design pattern for working on business-centric applications where each application is some unit of work on business data.

So if that's an actor, what is a virtual actor? The Virtual Actor pattern was first established by Microsoft Research in 2010.

[T]he Virtual Actor abstraction [...] provides a straightforward approach to building distributed interactive applications, without the need to learn complex programming patterns for handling concurrency, fault tolerance, and resource management.

How do you build an actor in such a way that it's considered virtual? In the original paper, virtual actors are defined by four criteria:

  1. Perpetual Existence
  2. Automatic Instantiation
  3. Location Transparency
  4. Automatic Scale-out

Perpetual existence requires an actor must always be available and ready to respond; automatic instantiation will create instances of actors as needed; location transparency states an actor's logical, virtual, or geographical location is not known; and automatic scale-out is defined by a virtual actor's ability to answer any request coming in.

The reference implementation, to which I am an occasional contributor (wish I had time for more), Orleans, is a .NET Foundation project for distributed virtual actors. It's battle-tested in production with Halo 4, Halo 5, and Gears of War as the core consumers, with Azure Service Fabric built on top of it. It's a great project with a very bright future, and I'm excited to see where it goes.

Until the 3.0 release, Orleans did not have external language support as all of it's distrusted messaging was using .NET-specific serialization. Now, there is more formal support for protocol buffers, meaning a cross-language interface is available, if you wanted to integrate other languages with Orleans. This is all well and great for .NET folks, but does Go have anything like that? Yes and no. There are open source variants leveraging the actor design pattern, but the question being are they really virtual and is it worth consuming and external library? Maybe. Go is designed around the concept of Communicating Sequential Processes, so all of the core pieces to build an actor pattern exist with the standard library. So let's do that!

In this example, I'll demonstrate how to build two virtual actors underneath a manager with a closed, private implementation. The code is available on Github. I'll call out fundamental design differences in my stdlib implementation against Orleans, mostly because I elected for a more idiomatic approach as it made more sense. Just because I could do it the same way doesn't mean it'd be easy to understand or implement due to fundamental language design differences.

This implementation has a virtual actor for adding two numbers together, a virtual actor for subtracting two numbers, and a virtual actor manager.

Design

Messaging

As the core requirement for the actor pattern is messaging, we want to ensure we pass messages back and forth instead of calling functions. The benefit of this is the API surface is abstracted away from us, so we can add, remove, or modify the message structure as needed without a need for an API overhaul. In this implementation, I leverage message passing instead of interface invocation just because it's more idiomatic Go, but this design pattern could be used with interfaces if you wanted to go to the effort.

Generally, I like to have two message definitions per action I need to invoke: Request and Response. The Request message contains all the relevant information for the virtual actor to execute it's action, as well as a reference to the Response, so the actor could pass a response back to the client. Again, this is a bit different than how Orleans does it - it's just more idiomatic.

// Request an Addition action
type AdditionVirtualActorRequest struct {
	X, Y     int
	Response chan *AdditionVirtualActorResponse
}

// Response to an Addition action.
type AdditionVirtualActorResponse struct {
	Result int
	Error  error
	Ok     bool
}

The AdditionVirtualActorRequest structure provides a straightforward way to pass your operands and receive a response, AdditionVirtualActorResponse. In this case, we use a pointer and pass around a pointer so we can pass by reference instead of by value. In our example, it's not a distributed implementation, so passing by reference is fine. If we were to build this out in a distributed fashion, it's likely we'd want to pass by value if we cared about nil fields.

In the response, we have the result we care about, any errors (it's addition, so this is just an example pattern), and an Ok moniker. The moniker is really something I personally care to implement, because I like the !ok pattern maps implement, so I can do if !resp.Okay { // handle }, not that it's better or required. The response is generally designed to be received over a channel in my implementation.

Using channels instead of just pointers is a little more robust design detail, because you can either pass a nil channel for no response, create a response channel and immediately block, or create a channel and then do other things while you wait on the response. I'll dive into more examples of this in a bit.

The subtraction virtual actor message pattern is nearly identical.

type SubtractionVirtualActorRequest struct {
	X, Y     int
	Response chan *SubtractionVirtualActorResponse
}

type SubtractionVirtualActorResponse struct {
	Result int
	Error  error
	Ok     bool
}

At this point, you're likely wondering why I'm using the same exact fields when I could use a base reference and embed it. I could, and there is a use case in another project in which I do, but the downside is how it comes to instantiation, which I'll get to in a bit.

Virtual Actors

I personally treat virtual actors a bit more high level in that a virtual actor can handle multiple message types, which means there are multiple implementation details behind the scenes. So generally I have a manager, a way to configure it, and then the virtualization aspect of it.

In the case of Go, virtualization isn't quite the same way that you would think of the virtual keyword in C# as Go doesn't support inheritance, derivation, or polymorphism. I quantify the implementation as virtual instead of a standard actor pattern because it meets the [an] actor always exists, virtually. It cannot be explicitly created or destroyed design facet in addition to the four criteria laid out in the research paper. In this specific demo, because it's not distributed, it doesn't meet the location transparency criteria, but that's okay because adding in location transparency is easy once you have a foundation for distribution.

Defining our addition virtual actor management layer looks like this:

// Settings for AdditionVirtualActor.
type AdditionVirtualActorSettings struct {
	QueueDepth int
	Autostart  bool
}

// The AdditionVirtualActor reference type.
type AdditionVirtualActor struct {
	RequestRouter chan interface{}
	Running bool

	logger                       *log.Logger
	additionVirtualActorListener chan AdditionVirtualActorRequest
}

It's a pretty straightforward implementation as well as configuration. Since we're passing messages, we need a QueueDepth to define how we handle channel configurations and an Autostart. The Autostart is really a way to surface a hook for testing purposes, you wouldn't want this functionality in a large project because you would want the implementation to autostart itself.

The AdditionVirtualActor is implements a RequestRouter, which is really just a "queue" of unknown types that we're going to route to their respective internal virtual actors. We have a flag for Running, mostly for test hooks, status checks, etc., an internal logger, and a channel for handling the incoming AdditionVirtualActorRequest messages.

Composing the settings into the addition virtual actor management layer is also nominal.

func NewAdditionVirtualActor(settings *AdditionVirtualActorSettings) (*AdditionVirtualActor, error) {
	a := &AdditionVirtualActor{
		RequestRouter:                make(chan interface{}, settings.QueueDepth),
		logger:                       log.New(os.Stdout, "addition-virtual-actor ", log.LstdFlags),
		additionVirtualActorListener: make(chan AdditionVirtualActorRequest, settings.QueueDepth),
	}

	if settings.Autostart {
		a.Run()
	}

	return a, nil
}

The entire public API surface on the very small, as you can see in the godocs. Now that we have the management layer, we need to ensure we start the process of adding a virtual actor and all that entails. Firstly, we have a panic deferrer to ensure we catch panics.

func (a *AdditionVirtualActor) deferrer() {
	if r := recover(); r != nil {
		a.logger.Printf("fatality!\n")
	}
}

Now we implement the first aspect of virtualization. Remember, virtualization just means we know nothing about the underlying implementation or it's lifecycle in this instance.

func (a *AdditionVirtualActor) Run() {
	go func() {
		defer a.deferrer()

		a.Running = true

		for {
			msg := <-a.RequestRouter
			switch msg.(type) {
			case AdditionVirtualActorRequest:
				a.additionVirtualActorListener <- msg.(AdditionVirtualActorRequest)
			}
		}
	}()

	a.additionRunner()
}

We have an anonymous function which is really a middleware message router. Since my virtual actor pattern generally supports multiple different yet related message types, I have a middleware message router so I can send it to the right internal virtual actor to handle the message. You could also add as many hooks and logic here as desired, since this is an example, there is nothing, but it's a very easy, extensible pattern to leverage.

Let's take a look at our actual internal virtual actor implementation.

func (a *AdditionVirtualActor) additionRunner() {
	go func() {
		defer a.deferrer()
		for {
			msg := <- a.additionVirtualActorListener
			go a.add(msg)
		}
	}()
}

func (a *AdditionVirtualActor) add(request AdditionVirtualActorRequest) {
	defer a.deferrer()

	sum := request.X + request.Y

	a.logger.Printf("sum %d + %d = %d", request.X, request.Y, sum)

	if request.Response != nil {
		request.Response <- &AdditionVirtualActorResponse{
			Result: request.X + request.Y,
			Error:  nil,
			Ok:     true,
		}
	}
}

We have a runner which reads messages from the virtual actor listener channel (where addition requests actually end up) and then it calls our add method. What makes it virtual is the AdditionVirtualActor.add lifecycle is 100% unmanaged and uncontrollable beyond instantiation. The internal virtual actor is created at invocation and is removed during garbage collection. In Orleans, this would be considered a Stateless Worker in that there is no virtual actor-specific state being stored and then recalled via an identity.

if request.Response != nil {
	request.Response <- &AdditionVirtualActorResponse{
		Result: request.X + request.Y,
		Error:  nil,
		Ok:     true,
	}
}

If you look at the last bit of the method, you'll notice I'm backreferencing the instantiation of the response struct. While not strictly required, I instantiate the response struct when the response is ready to be sent because I don't really need to operate on the response. This is a good design pattern to use because you shouldn't be performing transformations on an assigned response instance as that defeats the purpose of message passing with a data-oriented model. The data model in the message should be clean, concise, and perfunct. Anything else starts to get you into a risky place because of the cognitive and logical overhead required to keep track of the data model.

This is also why I don't use embedded response models in this implementation, as a response would look like this:

if request.Response != nil {
	request.Response <- &AdditionVirtualActorResponse{
		Response{
			Result: request.X + request.Y,
			Error:  nil,
			Ok:     true,
		},
	}
}

It feels like an unnecessary verbosity for the most part. This is also personal preference as I like my message models to be concise. Do what works.

I'm not going to dive into the subtraction virtual actor as it's essentially identical, but here it is for reference.

// Settings for SubtractionVirtualActor.
type SubtractionVirtualActorSettings struct {
	QueueDepth int
	Autostart  bool
}

// The SubtractionVirtualActor reference type.
type SubtractionVirtualActor struct {
	RequestRouter chan interface{}
	Running bool

	logger                          *log.Logger
	subtractionVirtualActorListener chan SubtractionVirtualActorRequest
}

func NewSubtractionVirtualActor(settings *SubtractionVirtualActorSettings) (*SubtractionVirtualActor, error) {
	a := &SubtractionVirtualActor{
		RequestRouter:                   make(chan interface{}, settings.QueueDepth),
		logger:                          log.New(os.Stdout, "subtraction-virtual-actor ", log.LstdFlags),
		subtractionVirtualActorListener: make(chan SubtractionVirtualActorRequest, settings.QueueDepth),
	}

	if settings.Autostart {
		a.Run()
	}

	return a, nil
}

func (s *SubtractionVirtualActor) deferrer() {
	if r := recover(); r != nil {
		s.logger.Printf("fatality!\n")
	}
}

func (s *SubtractionVirtualActor) Run() {
	go func() {
		defer s.deferrer()

		s.Running = true

		for {
			msg := <-s.RequestRouter
			switch msg.(type) {
			case SubtractionVirtualActorRequest:
				s.subtractionVirtualActorListener <- msg.(SubtractionVirtualActorRequest)
			}
		}
	}()

	s.subtractionRunner()
}

type SubtractionVirtualActorRequest struct {
	X, Y     int
	Response chan *SubtractionVirtualActorResponse
}

type SubtractionVirtualActorResponse struct {
	Result int
	Error  error
	Ok     bool
}

func (s *SubtractionVirtualActor) subtractionRunner() {
	go func() {
		defer s.deferrer()
		for {
			go s.subtract(<-s.subtractionVirtualActorListener)
		}
	}()
}

func (s *SubtractionVirtualActor) subtract(request SubtractionVirtualActorRequest) {
	defer s.deferrer()

	sum := request.X - request.Y

	s.logger.Printf("sum %d - %d = %d", request.X, request.Y, sum)

	if request.Response != nil {
		request.Response <- &SubtractionVirtualActorResponse{
			Result: request.X - request.Y,
			Error:  nil,
			Ok:     true,
		}
	}
}

Virtual Actor Manager

In my design and implementation, the virtual actor manager is the top-tier management layer, a single interface for interacting with our virtual actors.

type VirtualActorManagerSettings struct {
	QueueDepth                      int
	Autostart                       bool
	AdditionVirtualActorSettings    *adder.AdditionVirtualActorSettings
	SubtractionVirtualActorSettings *subtractor.SubtractionVirtualActorSettings
}

type VirtualActorManager struct {
	RequestRouter chan VirtualActorRequest
	Running bool

	logger                  *log.Logger
	additionVirtualActor    *adder.AdditionVirtualActor
	subtractionVirtualActor *subtractor.SubtractionVirtualActor
}

For the most part, it's just a container for handling virtual actors. The instantiation is more or less the same, it just calls our virtual actor implementations when it starts.

func NewVirtualActorManager(settings *VirtualActorManagerSettings) (*VirtualActorManager, error) {
	v := &VirtualActorManager{
		RequestRouter: make(chan VirtualActorRequest, settings.QueueDepth),
		logger:        log.New(os.Stdout, "virtual-actor-manager ", log.LstdFlags),
	}

	var err error

	v.additionVirtualActor, err = adder.NewAdditionVirtualActor(settings.AdditionVirtualActorSettings)
	if err != nil {
		return &VirtualActorManager{}, err
	}

	v.subtractionVirtualActor, err = subtractor.NewSubtractionVirtualActor(settings.SubtractionVirtualActorSettings)
	if err != nil {
		return &VirtualActorManager{}, err
	}

	if settings.Autostart {
		v.Run()
	}

	return v, nil
}

It also has a deferrer implementation:

func (v *VirtualActorManager) deferrer() {
	if r := recover(); r != nil {
		v.logger.Printf("fatality!\n")
	}
}

And a Run implementation:

type VirtualActorRequestType string

const (
	AdditionRequest    VirtualActorRequestType = "add"
	SubtractionRequest VirtualActorRequestType = "subtract"
)

func (v *VirtualActorManager) Run() {
	go func() {
		defer v.deferrer()

		v.Running = true

		for {
			msg := <-v.RequestRouter

			v.logger.Printf("received %s", msg.Type)

			switch msg.Type {
			case AdditionRequest:
				v.additionVirtualActor.RequestRouter <- msg.Payload
			case SubtractionRequest:
				v.subtractionVirtualActor.RequestRouter <- msg.Payload
			}
		}
	}()
}

Now, it's important to be aware that the message routing is a bit different here, because we are using a different message pattern.

type VirtualActorRequest struct {
	Type    VirtualActorRequestType
	Payload interface{}
}

The message here is really about the Type of virtual actor we want to instantiate and the Payload we want to send it. The router middleware really just determines how the message payload gets to the right virtual actor.

And that's it! That's how you create and manage virtual actors in Go. Just like that. Let's take a look at the practical side of things, i.e. how you use it.

Usage

Let's get our virtual actor management layer configured, firstly.

    autoStart := true
    queueDepth := 10

    vaOpts := &VirtualActorManagerSettings{
        QueueDepth: 10,
        Autostart:  autoStart,
        AdditionVirtualActorSettings: &adder.AdditionVirtualActorSettings{
            QueueDepth: queueDepth,
            Autostart:  autoStart,
        },
        SubtractionVirtualActorSettings: &subtractor.SubtractionVirtualActorSettings{
            QueueDepth: queueDepth,
            Autostart:  autoStart,
        },
    }

Pretty straightfoward. Ultimately, the queue depth really doesn't matter because it just needs to be high enough that N callers aren't blocked. We autostart things just because it's not a test implementation, so things might as well autostart on their own. And then we build it:

	v, err := NewVirtualActorManager(vaOpts)
	if err != nil {
		panic(err)
	}

At this point, your virtual actor system is running! Let's send a message to the addition virtual actor asking to add two numbers together.

	// prepare the response channel.
	arespc := make(chan *adder.AdditionVirtualActorResponse, 1)

	// send the request.
	x, y := 1, 1
	v.RequestRouter <- VirtualActorRequest{
		Type: AdditionRequest,
		Payload: adder.AdditionVirtualActorRequest{
			X:        x,
			Y:        y,
			Response: arespc,
		},
	}

As we don't need the response immediately, we can continue full steam ahead with a request to our subtract two numbers from our subtraction virtual actor.

// prep, make, and wait for the response.
	srespc := make(chan *subtractor.SubtractionVirtualActorResponse, 1)
	x, y = 1, 1
	v.RequestRouter <- VirtualActorRequest{
		Type: SubtractionRequest,
		Payload: subtractor.SubtractionVirtualActorRequest{
			X:        x,
			Y:        y,
			Response: srespc,
		},
	}
	sresp := <-srespc
	close(srespc)

	if !sresp.Ok {
		fmt.Println(fmt.Errorf("subtractor error: %s", sresp.Error))
	}

Since we need the response before we do anything else, we create our response channel and immediately block until it returns, closing the channel afterwards. Now we think we're ready for the response from the addition virtual actor.

	// read the response.
	aresp := <-arespc
	close(arespc)

	if !aresp.Ok {
		fmt.Println(fmt.Errorf("adder error: %s", aresp.Error))
	}

And that's it, nice and simple. If you didn't care about the response (maybe it was something which should've happened in the background), then because we guarded sending our response on the channel with a nil check, then you could just have a nil value in the response field of the request.

Putting it all together looks something like this:

func main() {

	autoStart := true
	queueDepth := 10

	vaOpts := &VirtualActorManagerSettings{
		QueueDepth: 10,
		Autostart:  autoStart,
		AdditionVirtualActorSettings: &adder.AdditionVirtualActorSettings{
			QueueDepth: queueDepth,
			Autostart:  autoStart,
		},
		SubtractionVirtualActorSettings: &subtractor.SubtractionVirtualActorSettings{
			QueueDepth: queueDepth,
			Autostart:  autoStart,
		},
	}

	v, err := NewVirtualActorManager(vaOpts)
	if err != nil {
		panic(err)
	}

	// v.Run() is not needed because we're autostarting the virtual actor pool.

	// prepare the response channel.
	arespc := make(chan *adder.AdditionVirtualActorResponse, 1)

	// send the request.
	x, y := 1, 1
	v.RequestRouter <- VirtualActorRequest{
		Type: AdditionRequest,
		Payload: adder.AdditionVirtualActorRequest{
			X:        x,
			Y:        y,
			Response: arespc,
		},
	}

	// prep, make, and wait for the response.
	srespc := make(chan *subtractor.SubtractionVirtualActorResponse, 1)
	x, y = 1, 1
	v.RequestRouter <- VirtualActorRequest{
		Type: SubtractionRequest,
		Payload: subtractor.SubtractionVirtualActorRequest{
			X:        x,
			Y:        y,
			Response: srespc,
		},
	}
	sresp := <-srespc
	close(srespc)

	if !sresp.Ok {
		fmt.Println(fmt.Errorf("subtractor error: %s", sresp.Error))
	}

	v.logger.Printf("subtracted %d - %d = %d\n", x, y, sresp.Result)

	// other things can be done now since we'll just read from the channel when we need the response.
	time.Sleep(time.Second * 1)

	// read the response.
	aresp := <-arespc
	close(arespc)

	if !aresp.Ok {
		fmt.Println(fmt.Errorf("adder error: %s", aresp.Error))
	}

	v.logger.Printf("added %d + %d = %d\n", x, y, aresp.Result)
}

Notes

Compared to Orleans, this is an overly simplistic virtual actor implementation that doesn't support a distributed deployment, and that's okay. Adding in distributed constructs such as scheduling, clustering, and resiliency is non-trivial and there's a lot more thought and effort which needs to go into the design. Theoretically, I could add in distributed constructs with a bit of extra work, but that's a bit out of the scope of this post since it's really about focusing on the core design pattern.

Conclusion

As always please feel free to let me know what you think.