Neural Network: Save/Load should preserve intermediate values #53

josephjaspers · 2019-11-19T02:23:59Z

Currently when when saving a model only the weights are preserved, however the state should be preserved as well.

xinsuinizhuan · 2019-12-02T05:33:49Z

Are you optimizing your network structure?

josephjaspers · 2019-12-03T00:11:24Z

Hello!

Since talking, I have added Convolution, Maxpooling (cpu and gpu) with optimized versions for convolution.

LSTM is not optimized yet (though it already runs at reasonable speed, I believe it can be even faster).

I have been adding a large amount of error checking and static_asserts which should make using the library easier (as it should more often give easy to understand errors).

The save/load differences are limited to the LSTM, but I can try to fix them soon. (It currently works as expected for non LSTMs)

I can try to solve the save/load issues soon as they shouldn't be too hard to fix.

xinsuinizhuan · 2019-12-06T02:01:43Z

Then i hope solved this issues。

xinsuinizhuan · 2019-12-11T00:39:50Z

How about this issues?

josephjaspers · 2019-12-11T22:34:00Z

I'm working on changing the LSTM to an optimzied version so I'm holding off on fixing this issue till than. (Hopefully will be done by end of this weekend).

xinsuinizhuan · 2019-12-11T22:49:30Z

thank you very much! 在 2019-12-12 06:34:00，"Joseph Jaspers" <[email protected]> 写道： I'm working on changing the LSTM to an optimzied version so I'm holding off on fixing this issue till than. (Hopefully will be done by end of this weekend). I am also trying to change the interface of Layers to make the easier to use. (It will look more like keras) — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

xinsuinizhuan · 2019-12-15T07:13:23Z

when i save the trained model parameters, my screen black and flicker,how to avoid it?

josephjaspers · 2019-12-15T17:36:40Z

Are you using the GPU implementation?

It most likely means that you are close to running out of (gpu) ram, if this happens with the CPU that is very strange, and I am not sure if this code could even cause such an issue. (Thought maybe a ram issue as well)

josephjaspers · 2019-12-15T17:38:35Z

Branch for the optimized LSTM
https://github.com/josephjaspers/blackcat_tensors/tree/optimize_lstm

Oddly enough it is faster in CPU mode but slower on the GPU (due to strided memory access).

I might need to still work on it as I would prefer not to have different versions for each architecture

josephjaspers · 2019-12-15T18:07:22Z

when i save the trained model parameters, my screen black and flicker,how to avoid it?

I just added a change to save/load (really the "to_string" implementation) that avoids a gpu-allocation when the memory is continuous.

This doesn't effect the CPU version at all.

cff6351

Additionally I have just added "clear_recycler"
BC::allocators::Recycle_Allocator_Globals::clear_recycler(system_tag);

Which will clear all currently unused memory in the Recycler_Allocator (The Neural Network doesn't deallocate its memory after use, and instead stores it in a global-dictionary for reuse).

This may help if you are close to running out of ram, however I would try if the latest code fixes your problem before trying this.

10a6f43 edit** minor fix: e724960

josephjaspers · 2019-12-16T00:38:30Z

Load/saving in LSTM has been fixed:
279581f

Caused by a bug with an incorrectly named method
"load_from_cache" -> "load_to_cache"

josephjaspers · 2019-12-16T00:39:27Z

You can test that saving/loading works by replacing the mnist_test_recurrent/mnist_test.h with:

#include "../../include/BlackCat_Tensors.h"
#include "../../include/BlackCat_NeuralNetworks.h"
#include "../datasets/mnist_loader.h"
#include <chrono>
#include <string>

template<class System=BC::host_tag>
int percept_MNIST(System system_tag, std::string mnist_dataset,
		int epochs=5, int batch_size=32, int samples=32*1024) {

	using value_type     = typename System::default_floating_point_type;
	using allocator_type = BC::Allocator<System, value_type>;
	using cube           = BC::Cube<value_type, allocator_type>;
	using mat            = BC::Matrix<value_type, allocator_type>;
	using clock          = std::chrono::duration<double>;

	auto network = BC::nn::neuralnetwork(
		BC::nn::lstm(system_tag, 784/4, 128),
		BC::nn::lstm(system_tag, 128, 64),
		BC::nn::feedforward(system_tag, 64, 10),
		BC::nn::softmax(system_tag, 10),
		BC::nn::logging_output_layer(system_tag, 10, BC::nn::RMSE).skip_every(100)
	);

	BC::print("Neural Network architecture:");
	BC::print(network.get_string_architecture());

	network.set_batch_size(batch_size);

	std::pair<cube, cube> data = load_mnist(
			system_tag, mnist_dataset, batch_size, samples);

	cube& inputs = data.first;
	cube& outputs = data.second;

	BC::print("training...");
	auto start = std::chrono::system_clock::now();

	int img_partitions = 4;
	for (int i = 0; i < epochs; ++i){
		BC::print("current epoch:  ", i);

		for (int j = 0; j < samples/batch_size; j++) {
			for (int p = 0; p < img_partitions; ++p) {

				auto batch = inputs[j];
				auto index = BC::index(0,784 * (p/(float)img_partitions));
				auto shape = BC::shape(784/4, batch_size);

				network.forward_propagation(batch[{index, shape}]);
			}
				//Apply backprop on the last two images (images 3/4 and 4/4)
				network.back_propagation(outputs[j]);
				network.back_propagation(outputs[j]);

				network.update_weights();
		}
	}


	auto end = std::chrono::system_clock::now();
	BC::print("training time:", clock(end - start).count());

	BC::print("testing...");

	network.save("mnist_test");

	auto batch = inputs[0];
	auto shape = BC::shape(784/4, batch_size);
	{
		for (int p = 0; p < img_partitions-1; ++p) {
			auto index = BC::index(0, 784*(p/(float)img_partitions));
			network.predict(batch[{index, shape}]);
		}

		auto last_index = BC::index(0,784*((img_partitions-1)/(float)img_partitions));
		mat hyps =network.predict(batch[{last_index, shape}]);

		BC::size_t test_images = 10;
		cube img = cube(inputs[0].reshaped(28,28, batch_size));
		for (int i = 0; i < test_images; ++i) {
			outputs[0][i];
			hyps[i].print();
			BC::print("------------------------------------");
		}
	}

	network.load("mnist_test");
	{
		for (int p = 0; p < img_partitions-1; ++p) {
			auto index = BC::index(0, 784*(p/(float)img_partitions));
			network.predict(batch[{index, shape}]);
		}

		auto last_index = BC::index(0,784*((img_partitions-1)/(float)img_partitions));
		mat hyps =network.predict(batch[{last_index, shape}]);

		BC::size_t test_images = 10;
		cube img = cube(inputs[0].reshaped(28,28, batch_size));
		for (int i = 0; i < test_images; ++i) {
			outputs[0][i];
			hyps[i].print();
			BC::print("------------------------------------");
		}
	}

	BC::print("success");
	return 0;
}

xinsuinizhuan · 2019-12-16T00:42:05Z

Are you using the GPU implementation?

It most likely means that you are close to running out of (gpu) ram, if this happens with the CPU that is very strange, and I am not sure if this code could even cause such an issue. (Thought maybe a ram issue as well)

No, i am so sorry, it seems that the console popup, and output something.

xinsuinizhuan · 2019-12-16T00:44:15Z

I only run it CPU，because using the GPU， the time is nearly equal to the cpu.

xinsuinizhuan · 2019-12-16T01:05:20Z

incorrectly named method

OK. Thank you very much, let me have a try!

xinsuinizhuan · 2019-12-16T02:04:28Z

You can test that saving/loading works by replacing the mnist_test_recurrent/mnist_test.h with:

#include "../../include/BlackCat_Tensors.h"
#include "../../include/BlackCat_NeuralNetworks.h"
#include "../datasets/mnist_loader.h"
#include <chrono>
#include <string>

template<class System=BC::host_tag>
int percept_MNIST(System system_tag, std::string mnist_dataset,
		int epochs=5, int batch_size=32, int samples=32*1024) {

	using value_type     = typename System::default_floating_point_type;
	using allocator_type = BC::Allocator<System, value_type>;
	using cube           = BC::Cube<value_type, allocator_type>;
	using mat            = BC::Matrix<value_type, allocator_type>;
	using clock          = std::chrono::duration<double>;

	auto network = BC::nn::neuralnetwork(
		BC::nn::lstm(system_tag, 784/4, 128),
		BC::nn::lstm(system_tag, 128, 64),
		BC::nn::feedforward(system_tag, 64, 10),
		BC::nn::softmax(system_tag, 10),
		BC::nn::logging_output_layer(system_tag, 10, BC::nn::RMSE).skip_every(100)
	);

	BC::print("Neural Network architecture:");
	BC::print(network.get_string_architecture());

	network.set_batch_size(batch_size);

	std::pair<cube, cube> data = load_mnist(
			system_tag, mnist_dataset, batch_size, samples);

	cube& inputs = data.first;
	cube& outputs = data.second;

	BC::print("training...");
	auto start = std::chrono::system_clock::now();

	int img_partitions = 4;
	for (int i = 0; i < epochs; ++i){
		BC::print("current epoch:  ", i);

		for (int j = 0; j < samples/batch_size; j++) {
			for (int p = 0; p < img_partitions; ++p) {

				auto batch = inputs[j];
				auto index = BC::index(0,784 * (p/(float)img_partitions));
				auto shape = BC::shape(784/4, batch_size);

				network.forward_propagation(batch[{index, shape}]);
			}
				//Apply backprop on the last two images (images 3/4 and 4/4)
				network.back_propagation(outputs[j]);
				network.back_propagation(outputs[j]);

				network.update_weights();
		}
	}


	auto end = std::chrono::system_clock::now();
	BC::print("training time:", clock(end - start).count());

	BC::print("testing...");

	network.save("mnist_test");

	auto batch = inputs[0];
	auto shape = BC::shape(784/4, batch_size);
	{
		for (int p = 0; p < img_partitions-1; ++p) {
			auto index = BC::index(0, 784*(p/(float)img_partitions));
			network.predict(batch[{index, shape}]);
		}

		auto last_index = BC::index(0,784*((img_partitions-1)/(float)img_partitions));
		mat hyps =network.predict(batch[{last_index, shape}]);

		BC::size_t test_images = 10;
		cube img = cube(inputs[0].reshaped(28,28, batch_size));
		for (int i = 0; i < test_images; ++i) {
			outputs[0][i];
			hyps[i].print();
			BC::print("------------------------------------");
		}
	}

	network.load("mnist_test");
	{
		for (int p = 0; p < img_partitions-1; ++p) {
			auto index = BC::index(0, 784*(p/(float)img_partitions));
			network.predict(batch[{index, shape}]);
		}

		auto last_index = BC::index(0,784*((img_partitions-1)/(float)img_partitions));
		mat hyps =network.predict(batch[{last_index, shape}]);

		BC::size_t test_images = 10;
		cube img = cube(inputs[0].reshaped(28,28, batch_size));
		for (int i = 0; i < test_images; ++i) {
			outputs[0][i];
			hyps[i].print();
			BC::print("------------------------------------");
		}
	}

	BC::print("success");
	return 0;
}

Did not to need network.copy_training_data_to_single_predict(0)？？

xinsuinizhuan · 2019-12-16T06:38:11Z

Neural Network: Load problem.
when i trained, i use the model to single_predict, the same input the same output, time after time. But when i load the model, then to single_predict, the same input, output is different, the different time. When i load the model, whether need to network.copy_training_data_to_single_predict(0)???

xinsuinizhuan · 2019-12-16T06:42:24Z

When i save the model, how to forbid the console popup to print something? Because i did not the console project, but when i save the model, the console popup and screen flicker.

xinsuinizhuan · 2019-12-17T01:50:44Z

mnist_test_example:
network.save("mnist_test");
auto batch = inputs[0];
BC::print("predict ------------------------------------");
auto shape = BC::shape(784 / 4, batch_size);
{
for (int p = 0; p < img_partitions - 1; ++p) {
auto index = BC::index(0, 784 * (p / (float)img_partitions));
network.predict(batch[{index, shape}]);
}

	auto last_index = BC::index(0, 784 * ((img_partitions - 1) / (float)img_partitions));
	mat hyps = network.predict(batch[{last_index, shape}]);

	BC::size_t test_images = 10;
	cube img = cube(inputs[0].reshaped(28, 28, batch_size));
	for (int i = 0; i < test_images; ++i) {
		outputs[0][i];
		hyps[i].print();
		BC::print("------------------------------------");
	}
}
    network.load("mnist_test");
BC::print("load fist ------------------------------------");
{
	for (int p = 0; p < img_partitions - 1; ++p) {
		auto index = BC::index(0, 784 * (p / (float)img_partitions));
		network.predict(batch[{index, shape}]);
	}

	auto last_index = BC::index(0, 784 * ((img_partitions - 1) / (float)img_partitions));
	mat hyps = network.predict(batch[{last_index, shape}]);

	BC::size_t test_images = 10;
	cube img = cube(inputs[0].reshaped(28, 28, batch_size));
	for (int i = 0; i < test_images; ++i) {
		outputs[0][i];
		hyps[i].print();
		BC::print("------------------------------------");
	}
}

BC::print("load second ------------------------------------");
{
	for (int p = 0; p < img_partitions - 1; ++p) {
		auto index = BC::index(0, 784 * (p / (float)img_partitions));
		network.predict(batch[{index, shape}]);
	}

	auto last_index = BC::index(0, 784 * ((img_partitions - 1) / (float)img_partitions));
	mat hyps = network.predict(batch[{last_index, shape}]);

	BC::size_t test_images = 10;
	cube img = cube(inputs[0].reshaped(28, 28, batch_size));
	for (int i = 0; i < test_images; ++i) {
		outputs[0][i];
		hyps[i].print();
		BC::print("------------------------------------");
	}
}

the reuslt:
when i load to predict, the result is not same as the load fist, and the load seconrd is not as all of them

josephjaspers · 2019-12-17T04:33:21Z

mnist_test_example:
network.save("mnist_test");
auto batch = inputs[0];
BC::print("predict ------------------------------------");
auto shape = BC::shape(784 / 4, batch_size);
{
for (int p = 0; p < img_partitions - 1; ++p) {
auto index = BC::index(0, 784 * (p / (float)img_partitions));
network.predict(batch[{index, shape}]);
}

	auto last_index = BC::index(0, 784 * ((img_partitions - 1) / (float)img_partitions));
	mat hyps = network.predict(batch[{last_index, shape}]);

	BC::size_t test_images = 10;
	cube img = cube(inputs[0].reshaped(28, 28, batch_size));
	for (int i = 0; i < test_images; ++i) {
		outputs[0][i];
		hyps[i].print();
		BC::print("------------------------------------");
	}
}
    network.load("mnist_test");
BC::print("load fist ------------------------------------");
{
	for (int p = 0; p < img_partitions - 1; ++p) {
		auto index = BC::index(0, 784 * (p / (float)img_partitions));
		network.predict(batch[{index, shape}]);
	}

	auto last_index = BC::index(0, 784 * ((img_partitions - 1) / (float)img_partitions));
	mat hyps = network.predict(batch[{last_index, shape}]);

	BC::size_t test_images = 10;
	cube img = cube(inputs[0].reshaped(28, 28, batch_size));
	for (int i = 0; i < test_images; ++i) {
		outputs[0][i];
		hyps[i].print();
		BC::print("------------------------------------");
	}
}

BC::print("load second ------------------------------------");
{
	for (int p = 0; p < img_partitions - 1; ++p) {
		auto index = BC::index(0, 784 * (p / (float)img_partitions));
		network.predict(batch[{index, shape}]);
	}

	auto last_index = BC::index(0, 784 * ((img_partitions - 1) / (float)img_partitions));
	mat hyps = network.predict(batch[{last_index, shape}]);

	BC::size_t test_images = 10;
	cube img = cube(inputs[0].reshaped(28, 28, batch_size));
	for (int i = 0; i < test_images; ++i) {
		outputs[0][i];
		hyps[i].print();
		BC::print("------------------------------------");
	}
}

the reuslt:
when i load to predict, the result is not same as the load fist, and the load seconrd is not as all of them

I will have to check that tomorrow!
I just added some code which will hopefully deal with the console pop up.

xinsuinizhuan · 2019-12-17T06:16:26Z

In my project, when i load the trained model, then to single_predict, the first time the output is worse, the sencord, the third, the fourth and fifth, the output got better and better, and eventually stabilized。

josephjaspers · 2019-12-18T13:00:19Z

In my project, when i load the trained model, then to single_predict, the first time the output is worse, the sencord, the third, the fourth and fifth, the output got better and better, and eventually stabilized。

--

Could you show me the outputs you receiving? I can't seem to emulate the problem. (Are you using copy_training_data_to_single_predict?)

josephjaspers · 2019-12-18T13:00:59Z

When i save the model, how to forbid the console popup to print something? Because i did not the console project, but when i save the model, the console popup and screen flicker.

Also is this still an issue? Sorry I have not been able to test on Windows yet.

josephjaspers · 2019-12-19T01:29:25Z

(Not related to this issue)
I have renamed the directories and changed the BC namespace to bc

the directories should be included as such:

#include "blackcat/tensors.h"
#include "blackcat/neural_networks.h"

I did this to be more consistent with most C++ style guides

xinsuinizhuan · 2019-12-19T02:21:23Z

(Not related to this issue)
I have renamed the directories and changed the BC namespace to bc

the directories should be included as such:
#include "blackcat/tensors.h"
#include "blackcat/neural_networks.h" 
I did this to be more consistent with most C++ style guides

Oh, No. when i use the new code, so many compile error:

xinsuinizhuan · 2019-12-19T02:35:09Z

When i save the model, how to forbid the console popup to print something? Because i did not the console project, but when i save the model, the console popup and screen flicker.

Also is this still an issue? Sorry I have not been able to test on Windows yet.

My net struct:
auto make_lstm_network() {
return BC::nn::neuralnetwork(
BC::nn::lstm(BC::host_tag(), 96 * 10, 1024, BC::nn::adam),
BC::nn::lstm(BC::host_tag(), 1024, 512, BC::nn::adam),
BC::nn::lstm(BC::host_tag(), 512, 216, BC::nn::adam),
BC::nn::feedforward(BC::host_tag(), 216, 192, BC::nn::adam),
BC::nn::logistic(BC::host_tag(), 192),
BC::nn::logging_output_layer(BC::host_tag(), 192, BC::nn::RMSE).skip_every(100)
);
}

my code:
//start train
LstmPredictTask* lstmpredicttask = new LstmPredictTask();
if (lstmpredicttask == NULL) {
return -2;
}

//LstmPredictTask lstmpredicttask;
std::cout << "Neural Network architecture: \n" << lstmpredicttask->m_pnetwork.get_string_architecture() << std::endl;
lstmpredicttask->m_pnetwork.set_learning_rate(lstmpredicttask->m_learning_rate);
lstmpredicttask->m_pnetwork.set_batch_size(lstmpredicttask->m_batch_size);

int training_sets;
std::pair<cube, cube> data = load_train_data(system_tag, datafilepath, lstmpredicttask, &training_sets);
cube& inputs = data.first;
cube& outputs = data.second;

std::cout <<" training..." << std::endl;
auto start = std::chrono::system_clock::now();

std::cout << "imagesinput real data:------------------------------------" << std::endl;
cube imagesinput = cube(inputs[0].reshaped(96, 10, lstmpredicttask->m_batch_size));
imagesinput[0].t().print_sparse();

std::cout << "imagesoutput real data:------------------------------------" << std::endl;
//auto imagesoutput = reshape(outputs[0], BC::shape(96, 2, lstmpredicttask->m_batch_size));
cube imagesoutput = cube(outputs[0].reshaped(96, 2, lstmpredicttask->m_batch_size));
imagesoutput[0].t().print_sparse();

    lstmpredicttask->m_pnetwork.load(_trainparamsavefile);

   std::cout << "process_predict_data first------------------------------------" << std::endl;
std::vector<double> forcast_outputvec;
process_predict_data(BC::host_tag(), lstmpredicttask, datafilepath, 18080, forcast_outputvec);
std::cout << "process_predict_data second------------------------------------" << std::endl;
std::vector<double> forcast_outputvec1;
process_predict_data(BC::host_tag(), lstmpredicttask, datafilepath, 18080, forcast_outputvec1);
std::cout << "process_predict_data third------------------------------------" << std::endl;
std::vector<double> forcast_outputvec2;
process_predict_data(BC::host_tag(), lstmpredicttask, datafilepath, 18080, forcast_outputvec2);

the single_predict result:

error.txt

i use the laod data to single_predict, use the same input, test three times, the first, second and the third time, the output is different. the result got better and better, the third time output close to the not load net single_predict.

josephjaspers · 2019-12-19T02:42:30Z

Looking into it now

josephjaspers · 2019-12-19T02:56:54Z

(Not related to this issue)
I have renamed the directories and changed the BC namespace to bc
the directories should be included as such:
#include "blackcat/tensors.h"
#include "blackcat/neural_networks.h" 
I did this to be more consistent with most C++ style guides
Oh, No. when i use the new code, so many compile error:

Hmm. I was able to compile without issues.
If you could send me the first errors I might be able to debug the issue.
It seems you are getting errors in cstdint which is strange as it is not my code.

xinsuinizhuan · 2019-12-19T03:11:20Z

(Not related to this issue)
I have renamed the directories and changed the BC namespace to bc
the directories should be included as such:
#include "blackcat/tensors.h"
#include "blackcat/neural_networks.h" 
I did this to be more consistent with most C++ style guides
Oh, No. when i use the new code, so many compile error:
Hmm. I was able to compile without issues.
If you could send me the first errors I might be able to debug the issue.
It seems you are getting errors in cstdint which is strange as it is not my code.

Yes, not your code, but old version code no problem, error:
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(27,14): error C2988: unrecognizable template declaration/definition
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(27,14): error C2143: syntax error: missing ';' before '<'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(27,14): error C2059: syntax error: '<'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(29,25): error C2143: syntax error: missing ';' before '{'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(29,25): error C2447: '{': missing function header (old-style formal list?)
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(70,14): error C2988: unrecognizable template declaration/definition
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(70,14): error C2143: syntax error: missing ';' before '<'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(70,14): error C2059: syntax error: '<'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(71,49): error C2143: syntax error: missing ';' before '{'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(71,49): error C2447: '{': missing function header (old-style formal list?)
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(76,14): error C2988: unrecognizable template declaration/definition
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(76,14): error C2143: syntax error: missing ';' before '<'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(76,14): error C2059: syntax error: '<'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(77,25): error C2143: syntax error: missing ';' before '{'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(77,25): error C2447: '{': missing function header (old-style formal list?)
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(82,14): error C2988: unrecognizable template declaration/definition
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(82,14): error C2143: syntax error: missing ';' before '<'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(82,14): error C2059: syntax error: '<'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(83,49): error C2143: syntax error: missing ';' before '{'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(83,49): error C2447: '{': missing function header (old-style formal list?)
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(89,29): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(89,29): error C2988: unrecognizable template declaration/definition
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(89,29): error C2143: syntax error: missing ',' before '&'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(92,24): error C2065: '_Str': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(95,43): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(95,50): error C2065: '_Base': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(97,17): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(98,9): error C3861: '_Xinvalid_argument': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(102,9): error C3861: '_Xout_of_range': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(105,9): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(106,10): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(106,37): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(112,30): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(112,30): error C2988: unrecognizable template declaration/definition
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(112,30): error C2143: syntax error: missing ',' before '&'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(115,24): error C2065: '_Str': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(118,43): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(118,50): error C2065: '_Base': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(120,17): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(121,9): error C3861: '_Xinvalid_argument': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(125,9): error C3861: '_Xout_of_range': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(128,9): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(129,10): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(129,37): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(135,40): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(135,40): error C2988: unrecognizable template declaration/definition
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(135,40): error C2143: syntax error: missing ',' before '&'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(138,24): error C2065: '_Str': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(141,53): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(141,60): error C2065: '_Base': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(143,17): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(144,9): error C3861: '_Xinvalid_argument': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(148,9): error C3861: '_Xout_of_range': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(151,9): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(152,10): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(152,37): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(158,36): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(158,36): error C2988: unrecognizable template declaration/definition
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(158,36): error C2143: syntax error: missing ',' before '&'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(161,24): error C2065: '_Str': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(164,49): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(164,56): error C2065: '_Base': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(166,17): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(167,9): error C3861: '_Xinvalid_argument': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(171,9): error C3861: '_Xout_of_range': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(174,9): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(175,10): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(175,37): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(181,46): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(181,46): error C2988: unrecognizable template declaration/definition
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(181,46): error C2143: syntax error: missing ',' before '&'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(184,24): error C2065: '_Str': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(187,59): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(187,66): error C2065: '_Base': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(189,17): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(190,9): error C3861: '_Xinvalid_argument': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(194,9): error C3861: '_Xout_of_range': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(197,9): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(198,10): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(198,37): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(204,31): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(204,31): error C2988: unrecognizable template declaration/definition
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(204,31): error C2143: syntax error: missing ',' before '&'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(206,24): error C2065: '_Str': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(209,44): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(211,17): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(212,9): error C3861: '_Xinvalid_argument': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(216,9): error C3861: '_Xout_of_range': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(219,9): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(220,10): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(220,37): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(226,32): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(226,32): error C2988: unrecognizable template declaration/definition
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(226,32): error C2143: syntax error: missing ',' before '&'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(228,24): error C2065: '_Str': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(231,45): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(233,17): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(234,9): error C3861: '_Xinvalid_argument': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(238,9): error C3861: '_Xout_of_range': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(241,9): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(242,10): error C2065: '_Idx': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(242,37): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(248,38): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(248,38): error C2988: unrecognizable template declaration/definition
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(248,38): error C2143: syntax error: missing ',' before '&'
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(250,24): error C2065: '_Str': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(253,51): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(255,17): error C2065: '_Eptr': undeclared identifier
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(256,9): error C3861: '_Xinvalid_argument': identifier not found
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(256,52): fatal error C1003: error count exceeds 100; stopping compilation
1>Done building project "MNIST_Test_Recurrent.vcxproj" -- FAILED.

josephjaspers · 2019-12-19T03:33:38Z

When I run this:
predict and single_predict give me the exact same results.

So I am not sure why your results aren't the same.
I think I would have to see how you are using single_predict to know what the issue is.

#include "../../blackcat/tensors.h"
#include "../../blackcat/neural_networks.h"
#include "../datasets/mnist_loader.h"
#include <chrono>
#include <string>

template<class System=bc::host_tag>
int percept_MNIST(System system_tag, std::string mnist_dataset,
		int epochs=5, int batch_size=32, int samples=32*1024) {

	using value_type     = typename System::default_floating_point_type;
	using allocator_type = bc::Allocator<System, value_type>;
	using cube           = bc::Cube<value_type, allocator_type>;
	using mat            = bc::Matrix<value_type, allocator_type>;
	using clock          = std::chrono::duration<double>;

	auto network = bc::nn::neuralnetwork(
		bc::nn::lstm(system_tag, 784/4, 128),
		bc::nn::lstm(system_tag, 128, 64),
		bc::nn::feedforward(system_tag, 64, 10),
		bc::nn::softmax(system_tag, 10),
		bc::nn::logging_output_layer(system_tag, 10, bc::nn::RMSE).skip_every(100)
	);

	bc::print("Neural Network architecture:");
	bc::print(network.get_string_architecture());

	network.set_batch_size(batch_size);

	std::pair<cube, cube> data = load_mnist(
			system_tag, mnist_dataset, batch_size, samples);

	cube& inputs = data.first;
	cube& outputs = data.second;

	bc::print("training...");
	auto start = std::chrono::system_clock::now();

	int img_partitions = 4;
	for (int i = 0; i < epochs; ++i){
		bc::print("current epoch:  ", i);

		for (int j = 0; j < samples/batch_size; j++) {
			for (int p = 0; p < img_partitions; ++p) {

				auto batch = inputs[j];
				auto index = bc::index(0,784 * (p/(float)img_partitions));
				auto shape = bc::shape(784/4, batch_size);

				network.forward_propagation(batch[{index, shape}]);
			}
				//Apply backprop on the last two images (images 3/4 and 4/4)
				network.back_propagation(outputs[j]);
				network.back_propagation(outputs[j]);

				network.update_weights();
		}
	}


	auto end = std::chrono::system_clock::now();
	bc::print("training time:", clock(end - start).count());

	bc::print("testing...");

	bc::print("saving..............");
	network.save("mnist_test");


	auto shape = bc::shape(784/4, batch_size);
	{

		bc::size_t test_images = 10;
		cube img = cube(inputs[0].reshaped(28,28, batch_size));
		for (int i = 0; i < test_images; ++i) {
			auto batch = inputs[i];
			for (int p = 0; p < img_partitions-1; ++p) {
						auto index = bc::index(0, 784*(p/(float)img_partitions));
						network.predict(batch[{index, shape}]);
					}

			auto last_index = bc::index(0,784*((img_partitions-1)/(float)img_partitions));
			mat hyps =network.predict(batch[{last_index, shape}]);

			outputs[i][0];
			hyps[0].print();
			bc::print("------------------------------------");
		}
	}

	bc::print("load1");
	network.load("mnist_test");
	network.copy_training_data_to_single_predict(0);
	{

		bc::size_t test_images = 10;
		cube img = cube(inputs[0].reshaped(28,28, batch_size));
		for (int i = 0; i < test_images; ++i) {
			auto batch = inputs[i];
			for (int p = 0; p < img_partitions-1; ++p) {
						auto index = bc::index(0, 784*(p/(float)img_partitions));
						network.single_predict(batch[{index, shape}][0]);
					}

			auto last_index = bc::index(0,784*((img_partitions-1)/(float)img_partitions));
			auto hyps = network.single_predict(batch[{last_index, shape}][0]);

			outputs[i][0];
			hyps.print();
			bc::print("------------------------------------");
		}
	}

	bc::print("load2");
	network.load("mnist_test");
	network.copy_training_data_to_single_predict(0);
	{

		bc::size_t test_images = 10;
		cube img = cube(inputs[0].reshaped(28,28, batch_size));
		for (int i = 0; i < test_images; ++i) {
			auto batch = inputs[i];
			for (int p = 0; p < img_partitions-1; ++p) {
						auto index = bc::index(0, 784*(p/(float)img_partitions));
						network.single_predict(batch[{index, shape}][0]);
					}

			auto last_index = bc::index(0,784*((img_partitions-1)/(float)img_partitions));
			auto hyps = network.single_predict(batch[{last_index, shape}][0]);

			outputs[i][0];
			hyps.print();
			bc::print("------------------------------------");
		}
	}

	bc::print("success");
	return 0;
}

xinsuinizhuan · 2019-12-19T03:38:07Z

When I run this:
predict and single_predict give me the exact same results.

So I am not sure why your results aren't the same.
I think I would have to see how you are using single_predict to know what the issue is.

#include "../../blackcat/tensors.h"
#include "../../blackcat/neural_networks.h"
#include "../datasets/mnist_loader.h"
#include <chrono>
#include <string>

template<class System=bc::host_tag>
int percept_MNIST(System system_tag, std::string mnist_dataset,
		int epochs=5, int batch_size=32, int samples=32*1024) {

	using value_type     = typename System::default_floating_point_type;
	using allocator_type = bc::Allocator<System, value_type>;
	using cube           = bc::Cube<value_type, allocator_type>;
	using mat            = bc::Matrix<value_type, allocator_type>;
	using clock          = std::chrono::duration<double>;

	auto network = bc::nn::neuralnetwork(
		bc::nn::lstm(system_tag, 784/4, 128),
		bc::nn::lstm(system_tag, 128, 64),
		bc::nn::feedforward(system_tag, 64, 10),
		bc::nn::softmax(system_tag, 10),
		bc::nn::logging_output_layer(system_tag, 10, bc::nn::RMSE).skip_every(100)
	);

	bc::print("Neural Network architecture:");
	bc::print(network.get_string_architecture());

	network.set_batch_size(batch_size);

	std::pair<cube, cube> data = load_mnist(
			system_tag, mnist_dataset, batch_size, samples);

	cube& inputs = data.first;
	cube& outputs = data.second;

	bc::print("training...");
	auto start = std::chrono::system_clock::now();

	int img_partitions = 4;
	for (int i = 0; i < epochs; ++i){
		bc::print("current epoch:  ", i);

		for (int j = 0; j < samples/batch_size; j++) {
			for (int p = 0; p < img_partitions; ++p) {

				auto batch = inputs[j];
				auto index = bc::index(0,784 * (p/(float)img_partitions));
				auto shape = bc::shape(784/4, batch_size);

				network.forward_propagation(batch[{index, shape}]);
			}
				//Apply backprop on the last two images (images 3/4 and 4/4)
				network.back_propagation(outputs[j]);
				network.back_propagation(outputs[j]);

				network.update_weights();
		}
	}


	auto end = std::chrono::system_clock::now();
	bc::print("training time:", clock(end - start).count());

	bc::print("testing...");

	bc::print("saving..............");
	network.save("mnist_test");


	auto shape = bc::shape(784/4, batch_size);
	{

		bc::size_t test_images = 10;
		cube img = cube(inputs[0].reshaped(28,28, batch_size));
		for (int i = 0; i < test_images; ++i) {
			auto batch = inputs[i];
			for (int p = 0; p < img_partitions-1; ++p) {
						auto index = bc::index(0, 784*(p/(float)img_partitions));
						network.predict(batch[{index, shape}]);
					}

			auto last_index = bc::index(0,784*((img_partitions-1)/(float)img_partitions));
			mat hyps =network.predict(batch[{last_index, shape}]);

			outputs[i][0];
			hyps[0].print();
			bc::print("------------------------------------");
		}
	}

	bc::print("load1");
	network.load("mnist_test");
	network.copy_training_data_to_single_predict(0);
	{

		bc::size_t test_images = 10;
		cube img = cube(inputs[0].reshaped(28,28, batch_size));
		for (int i = 0; i < test_images; ++i) {
			auto batch = inputs[i];
			for (int p = 0; p < img_partitions-1; ++p) {
						auto index = bc::index(0, 784*(p/(float)img_partitions));
						network.single_predict(batch[{index, shape}][0]);
					}

			auto last_index = bc::index(0,784*((img_partitions-1)/(float)img_partitions));
			auto hyps = network.single_predict(batch[{last_index, shape}][0]);

			outputs[i][0];
			hyps.print();
			bc::print("------------------------------------");
		}
	}

	bc::print("load2");
	network.load("mnist_test");
	network.copy_training_data_to_single_predict(0);
	{

		bc::size_t test_images = 10;
		cube img = cube(inputs[0].reshaped(28,28, batch_size));
		for (int i = 0; i < test_images; ++i) {
			auto batch = inputs[i];
			for (int p = 0; p < img_partitions-1; ++p) {
						auto index = bc::index(0, 784*(p/(float)img_partitions));
						network.single_predict(batch[{index, shape}][0]);
					}

			auto last_index = bc::index(0,784*((img_partitions-1)/(float)img_partitions));
			auto hyps = network.single_predict(batch[{last_index, shape}][0]);

			outputs[i][0];
			hyps.print();
			bc::print("------------------------------------");
		}
	}

	bc::print("success");
	return 0;
}

No, i aslo use the your mnist_text_recurrent example, the same isssue:
#53 (comment)

josephjaspers · 2019-12-19T05:07:10Z

Could you send me what process_predict_data is doing?

xinsuinizhuan · 2019-12-19T05:49:20Z

process_predict_data

this is process_predict_data function:

template
bool process_predict_data(System system_tag, LstmPredictTask* network, char* datafilepath, int _predictdate, std::vector& _outputvec)
{
using value_type = typename System::default_floating_point_type;
using allocator_type = BC::Allocator<System, value_type>;
using cube = BC::Cube<value_type, allocator_type>;
using mat = BC::Matrix<value_type, allocator_type>;
using vec = BC::Vector<value_type, allocator_type>;
using clock = std::chrono::duration;

//std::vector<pair<int, std::vector<double>>> normaliInputTimeSeries;
//std::vector<std::vector<double>> maxminSeries;
//std::vector<pair<int, std::vector<double>>> inputTimeSeries;
//vector<vector<double>> matricsInputSeries;
//readwholeline(datafilepath, inputTimeSeries, maxminSeries);
//wholeline_Normalization_maxmin(inputTimeSeries, maxminSeries, normaliInputTimeSeries);
////get_convertMatrixData(normaliInputTimeSeries, _predictdate, network->m_sequence_length, matricsInputSeries);
//get_convertBatchMatrixData(normaliInputTimeSeries, _predictdate, network->m_batch_size, network->m_sequence_length, matricsInputSeries);


std::vector<pair<int, std::vector<double>>> normaliInputTimeSeries;
double maxvalue = 0.0, minvalue = 0.0;
std::vector<pair<int, std::vector<double>>> inputTimeSeries;
vector<vector<double>> matricsInputSeries, matricsOutputSeries;
readwholeline_ex(datafilepath, inputTimeSeries, maxvalue, minvalue);
wholeline_Normalization_maxmin_ex(inputTimeSeries, maxvalue, minvalue, normaliInputTimeSeries);
get_convertMatrix_PredictData(normaliInputTimeSeries, _predictdate, network->m_sequence_length, matricsInputSeries);

if (matricsInputSeries.size() != 1)
{
	return false;
}

int img_sz = network->m_inputs_number * network->m_sequence_length;
if (matricsInputSeries[0].size() != img_sz)
{
	return false;
}

cube inputs(img_sz, 1, 1);
//tensor's memory is uninitialized
inputs.zero();
//inputs[0][0].data_format_convers(matricsInputSeries[0]);
BC::copy(inputs[0][0].get_stream(),matricsInputSeries[0].begin(), matricsInputSeries[0].end(), inputs[0][0].cw_begin());

vec hyps = network->m_pnetwork.single_predict(inputs[0][0]);

std::cout << "single_predict inputdata------------------------------------" << std::endl;
cube imagesinput0 = cube(inputs[0].reshaped(96, 10, 1));
imagesinput0[0].t().print_sparse();

std::cout << "single_predict output predict data------------------------------------" << std::endl;
hyps.print();


//copy data to vector
std::vector<double> predict_vec(hyps.size());
BC::copy(hyps.get_stream(), hyps.cw_begin(), hyps.cw_end(), predict_vec.begin());

//Assign 192 records data to two vector, every 96 record
std::vector<std::vector<double>> outTimeSeries;
for (int i = 0; i < predict_vec.size();)
{
	vector<double> output;
	for (int j = 0; j < network->m_inputs_number; j++)
	{
		output.push_back(predict_vec[i + j]);
	}
	outTimeSeries.push_back(output);
	i += network->m_inputs_number;
}

//data recove
std::vector<std::vector<double>> vec_recovedata;
//wholeline_recove_maxmin(outTimeSeries, maxminSeries, vec_recovedata);
wholeline_recove_maxmin_ex(outTimeSeries, maxvalue, minvalue, vec_recovedata);

//after recover, the merge the two vector data(every 96 record ) to one vector(192 data)
for (int k = 0; k < vec_recovedata.size(); k++)
{
	for (int h = 0; h < vec_recovedata[k].size(); h++)
	{
		_outputvec.push_back(vec_recovedata[k][h]);
	}
}

return true;

}

my whole example file:
blackcat_tensors_consoledemo.txt

josephjaspers · 2019-12-19T21:51:24Z

After

    lstmpredicttask->m_pnetwork.load(_trainparamsavefile);

you should call

    lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0);

Here:
network.load("mnist_test");
{
for (int p = 0; p < img_partitions-1; ++p) {
auto index = BC::index(0, 784*(p/(float)img_partitions));
network.predict(batch[{index, shape}]);
}

	auto last_index = BC::index(0,784*((img_partitions-1)/(float)img_partitions));
	mat hyps =network.predict(batch[{last_index, shape}]);

	BC::size_t test_images = 10;
	cube img = cube(inputs[0].reshaped(28,28, batch_size));
	for (int i = 0; i < test_images; ++i) {
		outputs[0][i];
		hyps[i].print();
		BC::print("------------------------------------");
	}
}

BC::print("success");
return 0;

}
Did not to need network.copy_training_data_to_single_predict(0)？？

I am using regular (batched) predict, so I do not need to copy the data over.

xinsuinizhuan · 2019-12-20T00:48:18Z

After
    lstmpredicttask->m_pnetwork.load(_trainparamsavefile);
you should call
    lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0);
Here:
network.load("mnist_test");
{
for (int p = 0; p < img_partitions-1; ++p) {
auto index = BC::index(0, 784*(p/(float)img_partitions));
network.predict(batch[{index, shape}]);
}
	auto last_index = BC::index(0,784*((img_partitions-1)/(float)img_partitions));
	mat hyps =network.predict(batch[{last_index, shape}]);

	BC::size_t test_images = 10;
	cube img = cube(inputs[0].reshaped(28,28, batch_size));
	for (int i = 0; i < test_images; ++i) {
		outputs[0][i];
		hyps[i].print();
		BC::print("------------------------------------");
	}
}

BC::print("success");
return 0;
}
Did not to need network.copy_training_data_to_single_predict(0)？？

I am using regular (batched) predict, so I do not need to copy the data over.

No, there are something wrong with it, it was still a bug. I add the lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0) after load, but in my project, after load, the same input, output three times still different.

//start train
LstmPredictTask* lstmpredicttask = new LstmPredictTask();
if (lstmpredicttask == NULL) {
	return -2;
}

//LstmPredictTask lstmpredicttask;
std::cout << "Neural Network architecture: \n" <<          lstmpredicttask->m_pnetwork.get_string_architecture() << std::endl;
lstmpredicttask->m_pnetwork.set_learning_rate(lstmpredicttask->m_learning_rate);
lstmpredicttask->m_pnetwork.set_batch_size(lstmpredicttask->m_batch_size);

int training_sets;
std::pair<cube, cube> data = load_train_data(system_tag, datafilepath, lstmpredicttask, &training_sets);
cube& inputs = data.first;
cube& outputs = data.second;

std::cout <<" training..." << std::endl;
auto start = std::chrono::system_clock::now();

std::cout << "imagesinput real data:------------------------------------" << std::endl;
cube imagesinput = cube(inputs[0].reshaped(96, 10, lstmpredicttask->m_batch_size));
imagesinput[0].t().print_sparse();

std::cout << "imagesoutput real data:------------------------------------" << std::endl;
cube imagesoutput = cube(outputs[0].reshaped(96, 2, lstmpredicttask->m_batch_size));
imagesoutput[0].t().print_sparse();

lstmpredicttask->m_pnetwork.load(_trainparamsavefile);
lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0);

std::cout << "process_predict_data first------------------------------------" << std::endl;
std::vector<double> forcast_outputvec;
process_predict_data(BC::host_tag(), lstmpredicttask, datafilepath, 18080, forcast_outputvec);
std::cout << "process_predict_data second------------------------------------" << std::endl;
std::vector<double> forcast_outputvec1;
process_predict_data(BC::host_tag(), lstmpredicttask, datafilepath, 18080, forcast_outputvec1);
std::cout << "process_predict_data third------------------------------------" << std::endl;
std::vector<double> forcast_outputvec2;
process_predict_data(BC::host_tag(), lstmpredicttask, datafilepath, 18080, forcast_outputvec2);

result:

xinsuinizhuan · 2019-12-21T06:36:51Z

I am so sorry.After i load the net , then add the lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0), to single_predict, the result is so closely to the no-load single_predict, so this is not still a bug. But when i use the new code, so many compile error, it still the bug, it seems confilict whit the system string, somewheres. And when i save the net, console still pop to print somethings.

xinsuinizhuan · 2019-12-21T10:36:28Z

when i to single_predict time by time, i think same input, and same output, it should only be use the trained parameters and not to update and change the parameters. I fell it still a bug, it it close to the no-load output, but time by time, the output is different.

josephjaspers · 2019-12-22T03:10:21Z

I am so sorry.After i load the net , then add the lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0), to single_predict, the result is so closely to the no-load single_predict, so this is not still a bug. But when i use the new code, so many compile error, it still the bug, it seems confilict whit the system string, somewheres. And when i save the net, console still pop to print somethings.

Glad to hear it works.

I will try fix the console pop up bug.

The new code compiles on Windows for me so I will have to look into that more.

xinsuinizhuan · 2019-12-23T01:15:26Z

I am so sorry.After i load the net , then add the lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0), to single_predict, the result is so closely to the no-load single_predict, so this is not still a bug. But when i use the new code, so many compile error, it still the bug, it seems confilict whit the system string, somewheres. And when i save the net, console still pop to print somethings.

Glad to hear it works.

I will try fix the console pop up bug.

The new code compiles on Windows for me so I will have to look into that more.

You have the blackcat\string.h file, and the struct name is same as the MVS's string file and class name:

struct string:
std::string {

so i think they are confict.

so when i rename the blackcat\string.h to blackcat\bcstring.h, and rename the struct string in blackcat\string.h to struct bcstring. and change the string name to bcstring in the io.h and filesystem.h file, it works.

josephjaspers · 2019-12-26T22:57:56Z

I am so sorry.After i load the net , then add the lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0), to single_predict, the result is so closely to the no-load single_predict, so this is not still a bug. But when i use the new code, so many compile error, it still the bug, it seems confilict whit the system string, somewheres. And when i save the net, console still pop to print somethings.

Glad to hear it works.
I will try fix the console pop up bug.
The new code compiles on Windows for me so I will have to look into that more.

You have the blackcat\string.h file, and the struct name is same as the MVS's string file and class name:

struct string:
std::string {

so i think they are confict.

so when i rename the blackcat\string.h to blackcat\bcstring.h, and rename the struct string in blackcat\string.h to struct bcstring. and change the string name to bcstring in the io.h and filesystem.h file, it works.

My string class is in the bc namespace so it should not have any issues.

Are you directly including the blackcat directory? As that could cause a an inclusion issue with the std:: library.

Try to include the folder containing blackcat opposed to including the blackcat folder directly.

Correct:
#include "blackcat/tensors.h"

Incorrect:
#include "tensors.h"

xinsuinizhuan · 2019-12-27T01:33:34Z

I am so sorry.After i load the net , then add the lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0), to single_predict, the result is so closely to the no-load single_predict, so this is not still a bug. But when i use the new code, so many compile error, it still the bug, it seems confilict whit the system string, somewheres. And when i save the net, console still pop to print somethings.

Glad to hear it works.
I will try fix the console pop up bug.
The new code compiles on Windows for me so I will have to look into that more.

You have the blackcat\string.h file, and the struct name is same as the MVS's string file and class name:
struct string:
std::string {
so i think they are confict.
so when i rename the blackcat\string.h to blackcat\bcstring.h, and rename the struct string in blackcat\string.h to struct bcstring. and change the string name to bcstring in the io.h and filesystem.h file, it works.

My string class is in the bc namespace so it should not have any issues.

Are you directly including the blackcat directory? As that could cause a an inclusion issue with the std:: library.

Try to include the folder containing blackcat opposed to including the blackcat folder directly.

Correct:
#include "blackcat/tensors.h"

Incorrect:
#include "tensors.h"

thank you very much. when i use #include "blackcat/tensors.h", it works.

josephjaspers · 2019-12-27T02:04:46Z

Awesome, glad it worked!

xinsuinizhuan · 2019-12-27T02:09:13Z

What's next plan? attention-lstm, gpu optimize?

josephjaspers · 2019-12-27T02:26:19Z

I want to write a few tests and examples first,
than I will do attention-lstm

josephjaspers mentioned this issue Nov 19, 2019

the forward_propagation、predict function batched forward and the signal_predict function, same input but predict output different. #45

Closed

xinsuinizhuan referenced this issue Dec 26, 2019

Remove dead or duplicate code

bfbcf8c

Neural Network: Save/Load should preserve intermediate values #53

Neural Network: Save/Load should preserve intermediate values #53

Comments

josephjaspers commented Nov 19, 2019

xinsuinizhuan commented Dec 2, 2019

josephjaspers commented Dec 3, 2019

xinsuinizhuan commented Dec 6, 2019

xinsuinizhuan commented Dec 11, 2019

josephjaspers commented Dec 11, 2019 • edited Loading

xinsuinizhuan commented Dec 11, 2019 via email

xinsuinizhuan commented Dec 15, 2019 • edited Loading

josephjaspers commented Dec 15, 2019

josephjaspers commented Dec 15, 2019

josephjaspers commented Dec 15, 2019 • edited Loading

josephjaspers commented Dec 16, 2019

josephjaspers commented Dec 16, 2019

xinsuinizhuan commented Dec 16, 2019

xinsuinizhuan commented Dec 16, 2019

xinsuinizhuan commented Dec 16, 2019

xinsuinizhuan commented Dec 16, 2019 • edited Loading

xinsuinizhuan commented Dec 16, 2019

xinsuinizhuan commented Dec 16, 2019 • edited Loading

xinsuinizhuan commented Dec 17, 2019

josephjaspers commented Dec 17, 2019

xinsuinizhuan commented Dec 17, 2019 • edited Loading

josephjaspers commented Dec 18, 2019

josephjaspers commented Dec 18, 2019

josephjaspers commented Dec 19, 2019 • edited Loading

xinsuinizhuan commented Dec 19, 2019

xinsuinizhuan commented Dec 19, 2019 • edited Loading

josephjaspers commented Dec 19, 2019

josephjaspers commented Dec 19, 2019

xinsuinizhuan commented Dec 19, 2019

josephjaspers commented Dec 19, 2019

xinsuinizhuan commented Dec 19, 2019

josephjaspers commented Dec 19, 2019

xinsuinizhuan commented Dec 19, 2019 • edited Loading

josephjaspers commented Dec 19, 2019

xinsuinizhuan commented Dec 20, 2019 • edited Loading

xinsuinizhuan commented Dec 21, 2019

xinsuinizhuan commented Dec 21, 2019

josephjaspers commented Dec 22, 2019

xinsuinizhuan commented Dec 23, 2019 • edited Loading

josephjaspers commented Dec 26, 2019

xinsuinizhuan commented Dec 27, 2019

josephjaspers commented Dec 27, 2019

xinsuinizhuan commented Dec 27, 2019

josephjaspers commented Dec 27, 2019 • edited Loading

josephjaspers commented Dec 11, 2019 •

edited

Loading

xinsuinizhuan commented Dec 15, 2019 •

edited

Loading

josephjaspers commented Dec 15, 2019 •

edited

Loading

xinsuinizhuan commented Dec 16, 2019 •

edited

Loading

xinsuinizhuan commented Dec 16, 2019 •

edited

Loading

xinsuinizhuan commented Dec 17, 2019 •

edited

Loading

josephjaspers commented Dec 19, 2019 •

edited

Loading

xinsuinizhuan commented Dec 19, 2019 •

edited

Loading

xinsuinizhuan commented Dec 19, 2019 •

edited

Loading

xinsuinizhuan commented Dec 20, 2019 •

edited

Loading

xinsuinizhuan commented Dec 23, 2019 •

edited

Loading

josephjaspers commented Dec 27, 2019 •

edited

Loading