Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2198 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.6925. Min Range: -2.6925

Max Act: 2.4215. Min Act: -0.0000

Data Index: 181867 (C4 (Web Text))

Max Activating Token Index: 313

Text #1

Max Range: 2.6925. Min Range: -2.6925

Max Act: 2.6396. Min Act: -0.0000

Data Index: 659038 (C4 (Web Text))

Max Activating Token Index: 680

Text #2

Max Range: 2.6925. Min Range: -2.6925

Max Act: 2.6925. Min Act: -0.0000

Data Index: 41211 (C4 (Web Text))

Max Activating Token Index: 808

Text #3

Max Range: 2.6925. Min Range: -2.6925

Max Act: 2.4292. Min Act: -0.0000

Data Index: 1330049 (C4 (Web Text))

Max Activating Token Index: 354

Text #4

Max Range: 2.6925. Min Range: -2.6925

Max Act: 2.2989. Min Act: -0.0000

Data Index: 1244458 (C4 (Web Text))

Max Activating Token Index: 625

Text #5

Max Range: 2.6925. Min Range: -2.6925

Max Act: 2.1701. Min Act: -0.0000

Data Index: 967058 (C4 (Web Text))

Max Activating Token Index: 710

Text #6

Max Range: 2.6925. Min Range: -2.6925

Max Act: 2.0917. Min Act: -0.0000

Data Index: 5841 (C4 (Web Text))

Max Activating Token Index: 57

Text #7

Max Range: 2.6925. Min Range: -2.6925

Max Act: 2.0480. Min Act: -0.0000

Data Index: 149286 (C4 (Web Text))

Max Activating Token Index: 363

Text #8

Max Range: 2.6925. Min Range: -2.6925

Max Act: 1.9252. Min Act: -0.0000

Data Index: 315538 (C4 (Web Text))

Max Activating Token Index: 926

Text #9

Max Range: 2.6925. Min Range: -2.6925

Max Act: 1.8873. Min Act: -0.0000

Data Index: 55849 (C4 (Web Text))

Max Activating Token Index: 260

Text #10

Max Range: 2.6925. Min Range: -2.6925

Max Act: 1.9093. Min Act: -0.0000

Data Index: 439045 (C4 (Web Text))

Max Activating Token Index: 475

Text #11

Max Range: 2.6925. Min Range: -2.6925

Max Act: 2.1347. Min Act: -0.0000

Data Index: 1016746 (C4 (Web Text))

Max Activating Token Index: 364

Text #12

Max Range: 2.6925. Min Range: -2.6925

Max Act: 1.8760. Min Act: -0.0000

Data Index: 862188 (C4 (Web Text))

Max Activating Token Index: 385

Text #13

Max Range: 2.6925. Min Range: -2.6925

Max Act: 1.9888. Min Act: -0.0000

Data Index: 57128 (C4 (Web Text))

Max Activating Token Index: 185

Text #14

Max Range: 2.6925. Min Range: -2.6925

Max Act: 1.9589. Min Act: -0.0000

Data Index: 503226 (C4 (Web Text))

Max Activating Token Index: 534

Text #15

Max Range: 2.6925. Min Range: -2.6925

Max Act: 1.8267. Min Act: -0.0000

Data Index: 1252016 (C4 (Web Text))

Max Activating Token Index: 271

Text #16

Max Range: 2.6925. Min Range: -2.6925

Max Act: 1.8414. Min Act: -0.0000

Data Index: 12759 (C4 (Web Text))

Max Activating Token Index: 281

Text #17

Max Range: 2.6925. Min Range: -2.6925

Max Act: 2.0240. Min Act: -0.0000

Data Index: 933791 (C4 (Web Text))

Max Activating Token Index: 888

Text #18

Max Range: 2.6925. Min Range: -2.6925

Max Act: 1.9386. Min Act: -0.0000

Data Index: 940144 (C4 (Web Text))

Max Activating Token Index: 639

Text #19

Max Range: 2.6925. Min Range: -2.6925

Max Act: 1.8837. Min Act: -0.0000

Data Index: 384508 (C4 (Web Text))

Max Activating Token Index: 585