Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 179 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2507. Min Act: -0.0001

Data Index: 191658 (C4 (Web Text))

Max Activating Token Index: 362

Text #1

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2357. Min Act: -0.0000

Data Index: 620499 (C4 (Web Text))

Max Activating Token Index: 777

Text #2

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2302. Min Act: -0.0000

Data Index: 550424 (C4 (Web Text))

Max Activating Token Index: 325

Text #3

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2265. Min Act: -0.0000

Data Index: 621339 (C4 (Web Text))

Max Activating Token Index: 432

Text #4

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2330. Min Act: -0.0000

Data Index: 195168 (C4 (Web Text))

Max Activating Token Index: 371

Text #5

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2461. Min Act: -0.0000

Data Index: 1306715 (C4 (Web Text))

Max Activating Token Index: 841

Text #6

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2205. Min Act: -0.0000

Data Index: 825465 (C4 (Web Text))

Max Activating Token Index: 524

Text #7

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2132. Min Act: -0.0000

Data Index: 10460 (C4 (Web Text))

Max Activating Token Index: 398

Text #8

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2229. Min Act: -0.0000

Data Index: 1166683 (C4 (Web Text))

Max Activating Token Index: 182

Text #9

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2208. Min Act: -0.0000

Data Index: 1022074 (C4 (Web Text))

Max Activating Token Index: 345

Text #10

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2329. Min Act: -0.0000

Data Index: 644547 (C4 (Web Text))

Max Activating Token Index: 318

Text #11

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2109. Min Act: -0.0000

Data Index: 300431 (C4 (Web Text))

Max Activating Token Index: 502

Text #12

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.1909. Min Act: -0.0000

Data Index: 571286 (C4 (Web Text))

Max Activating Token Index: 688

Text #13

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2071. Min Act: -0.0000

Data Index: 1122543 (C4 (Web Text))

Max Activating Token Index: 490

Text #14

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.1908. Min Act: -0.0000

Data Index: 355344 (C4 (Web Text))

Max Activating Token Index: 250

Text #15

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2028. Min Act: -0.0000

Data Index: 867523 (C4 (Web Text))

Max Activating Token Index: 468

Text #16

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.1779. Min Act: -0.0000

Data Index: 440768 (C4 (Web Text))

Max Activating Token Index: 360

Text #17

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.1870. Min Act: -0.0000

Data Index: 876700 (C4 (Web Text))

Max Activating Token Index: 289

Text #18

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2057. Min Act: -0.0000

Data Index: 271380 (C4 (Web Text))

Max Activating Token Index: 325

Text #19

Max Range: 0.2507. Min Range: -0.2507

Max Act: 0.2036. Min Act: -0.0000

Data Index: 585159 (C4 (Web Text))

Max Activating Token Index: 876