The Challenge

Now we have seen most building block which you need to start with the final challenge. The challenge is the 3d reconstruction of a real world face image. The data you get is only a single image of a face. In the following sections we provide you some more hints about what you can use to adapt the model to the image. Then it is up to you to design a good compilation of the building blocks to solve the task.

The Goal

The goal is to calculate a 3d reconstruction from a single 2d image. Two images are in the folder challenge from the project repository. The indoor image is in a more controlled setting than the outdoor image. The reconstruction of the second image is harder due to the more complex illumination, pose and the cluttered background. So we suggest to start with the easier image.

Minimal starting point

We first introduce some parts that are essential to get to a first running application before in the next section we introduce more advanced topics.

Landmark Clicker

You can use the landmark clicker (.jar-file) to annotate landmarks in the image. You can load your own list of landmarks by providing a valid TLMS-2d landmark file or simply use the predefined ones.

You then can load a *.tlms file using:

val landmarks = TLMSLandmarksIO.read2D(new File("path/to/file.tlms")).get

Tackle the image color

To adapt the model to the image more accurately you need to use the image colors. As for the computer graphics we need also illumination, you need to change this part of the render parameters too. You can use the following classes from the framework to handle colors:

To adapt the light there are two options. Either a update sampled from a Gaussian distribution can be used, or we can optimize the light parameters given the current state:

With these informations you should be able to already start fitting the image. Try to come up with an initial working algorithm. This will help you to develop a feeling for the problem and to select which part you would like to improve on.

We will sketch a lot of advanced topics you can address in the next section.

Advanced topics

Here we present advanced topics which arise when working on the reconstruction problem. Note that you do not have to address all these topics and you will not have enough time to tackle them all during this summer-school.

Miscellanous points

Proposals

When designing proposals the important decisions are:

Faster Evaluators

An important topic for more performance is caching. As we use a software renderer, the rendering takes more time compared to a hardware solution. We can overcome part of this problem using the MoMoRenderer's caching mechanism. You can take advantage of it by simply calling cached(n: Int) where n is the size of the cache:

val cachedRenderer: MoMoRenderer = MoMoRenderer(model).cached(10)

Further you can also cache the value of evaluators using the CachedDistributionEvaluator. This makes sense for costly evaluations as e.g. the image evaluator which iterates over all pixels in the image. You can cache an evaluator like this:

import scalismo.faces.sampling.evaluators.CachedDistributionEvaluator.implicits._

val cachedEvaluator = evaluator.cached(10)

Filtering

Another possibility to speed up the algorithm is to use step-wise Bayesian inference. Step-wise bayesian inference is not only about gaining speed. It is also useful to integrate different sources of information, as for example the landmarks and the information contained in the image.

In the software the bayesian inference or filtering is implemented using the MetropolisFilterProposal:

import scalismo.sampling.proposals.MetropolisFilterProposal

val proposal: SymmetricProposalGenerator[RenderParameter] = ???
val evaluator: DistributionEvaluator[RenderParameter] = ???
val filteredProposals = MetropolisFilterProposal(proposal,evaluator)

Loggers

Loggers provide a good starting point to get a deeper insight on a sampling run. Using an AcceptRejectLogger, you can print out the evolution of your sampling chain during the run in the following way:

val verboseLogger = new AcceptRejectLogger[RenderParameter] {
  var index = 0

  override def accept(current: RenderParameter,
                      sample: RenderParameter,
                      generator: ProposalGenerator[RenderParameter],
                      evaluator: DistributionEvaluator[RenderParameter]): Unit = {
    printMessage("A ",current,sample,generator,evaluator)
  }

  override def reject(current: RenderParameter,
                      sample: RenderParameter,
                      generator: ProposalGenerator[RenderParameter],
                      evaluator: DistributionEvaluator[RenderParameter]): Unit = {
    printMessage("R ",current,sample,generator,evaluator)
  }

    def printMessage(prefix: String,
                     current: RenderParameter,
                     sample: RenderParameter,
                     generator: ProposalGenerator[RenderParameter],
                     evaluator: DistributionEvaluator[RenderParameter]): Unit ={
      println(s"$prefix ${"%06d".format(index)} : ${evaluator.logValue(sample)}")
    }
  }

The AcceptRejectLogger has to be passed as second argument to the iterator function called on the sampling algorithm.

val algorithm = Metropolis(proposal,evaluator)
val chainIterator = algorithm.iterator(init,verboseLogger)

As an additional ChainStateLogger you can use one that keeps track of the best sample given an evaluator.

val bestLogger = BestSampleLogger(evaluator)
/* ...
 then you run your sampling application
 ... */
val bestSample = bestLogger.currentBestSample().get

When you want to add multiple loggers you have to combine them. Multiple loggers can be combined using logger containers:

import scalismo.sampling.loggers.ChainStateLoggerContainer
import scalismo.sampling.loggers.AcceptRejectLoggerContainer

val combinedCSLoggers = ChainStateLoggerContainer(Seq(csLoggerA, csLoggerB))
val combinedARLoggers = AcceptRejectLoggerContainer(Seq(arLoggerA, arLoggerB))

Image Likelihood Models

When we define the image likelihood model we have to model explicitly the background (see additional information). The best choice depends on the target image.

There are different image likelihood models you can choose from are:

Possible choices for the foreground model are:

Some options of the background model are:

In the end we want a sampling chain over RenderParameter and hence need a DistributionEvaluator[RenderParameter]. We can transform a PairEvaluator[PixelImage[RGBA]] to this type in two steps using the ImageRendererEvaluator and the toDistributionEvaluator function:

val distEval: DistributionEvaluator[PixelImage[RGBA]] = independentEval.toDistributionEvaluator(targetImage)
val imageEval = ImageRendererEvaluator(renderer = renderer, imageEvaluator = distEval)