Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4201 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 9.2070. Min Range: -9.2070

Max Act: 9.2070. Min Act: -0.0000

Data Index: 110337 (C4 (Web Text))

Max Activating Token Index: 499

Text #1

Max Range: 9.2070. Min Range: -9.2070

Max Act: 8.6384. Min Act: -0.0000

Data Index: 716804 (C4 (Web Text))

Max Activating Token Index: 641

Text #2

Max Range: 9.2070. Min Range: -9.2070

Max Act: 8.2861. Min Act: -0.0000

Data Index: 555771 (C4 (Web Text))

Max Activating Token Index: 566

Text #3

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.8232. Min Act: -0.0000

Data Index: 32080 (C4 (Web Text))

Max Activating Token Index: 470

Text #4

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.4699. Min Act: -0.0000

Data Index: 523794 (C4 (Web Text))

Max Activating Token Index: 335

Text #5

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.6521. Min Act: -0.0000

Data Index: 50726 (C4 (Web Text))

Max Activating Token Index: 471

Text #6

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.5913. Min Act: -0.0000

Data Index: 73111 (C4 (Web Text))

Max Activating Token Index: 392

Text #7

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.5894. Min Act: -0.0000

Data Index: 164715 (C4 (Web Text))

Max Activating Token Index: 403

Text #8

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.5476. Min Act: -0.0000

Data Index: 1151485 (C4 (Web Text))

Max Activating Token Index: 211

Text #9

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.5888. Min Act: -0.0000

Data Index: 983908 (C4 (Web Text))

Max Activating Token Index: 451

Text #10

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.3641. Min Act: -0.0000

Data Index: 607333 (C4 (Web Text))

Max Activating Token Index: 393

Text #11

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.3621. Min Act: -0.0000

Data Index: 1162036 (C4 (Web Text))

Max Activating Token Index: 283

Text #12

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.5339. Min Act: -0.0000

Data Index: 67332 (C4 (Web Text))

Max Activating Token Index: 456

Text #13

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.7173. Min Act: -0.0000

Data Index: 720303 (C4 (Web Text))

Max Activating Token Index: 318

Text #14

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.5346. Min Act: -0.0000

Data Index: 621595 (C4 (Web Text))

Max Activating Token Index: 408

Text #15

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.5295. Min Act: -0.0000

Data Index: 620837 (C4 (Web Text))

Max Activating Token Index: 303

Text #16

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.5475. Min Act: -0.0001

Data Index: 1331360 (C4 (Web Text))

Max Activating Token Index: 354

Text #17

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.7020. Min Act: -0.0000

Data Index: 793582 (C4 (Web Text))

Max Activating Token Index: 501

Text #18

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.5177. Min Act: -0.0000

Data Index: 800739 (C4 (Web Text))

Max Activating Token Index: 277

Text #19

Max Range: 9.2070. Min Range: -9.2070

Max Act: 7.4886. Min Act: -0.0000

Data Index: 339418 (C4 (Web Text))

Max Activating Token Index: 249