Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2497 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.7734. Min Act: -0.0000

Data Index: 1576994 (Python Code)

Max Activating Token Index: 886

Text #1

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.7472. Min Act: -0.0000

Data Index: 1584913 (Python Code)

Max Activating Token Index: 864

Text #2

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.7267. Min Act: -0.0000

Data Index: 1638213 (Python Code)

Max Activating Token Index: 943

Text #3

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.8544. Min Act: -0.0000

Data Index: 1521129 (Python Code)

Max Activating Token Index: 1018

Text #4

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.4536. Min Act: -0.0000

Data Index: 1380211 (Python Code)

Max Activating Token Index: 633

Text #5

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.5854. Min Act: 0.0000

Data Index: 1468554 (Python Code)

Max Activating Token Index: 416

Text #6

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.6909. Min Act: -0.0000

Data Index: 1609516 (Python Code)

Max Activating Token Index: 194

Text #7

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.8065. Min Act: -0.0000

Data Index: 1450605 (Python Code)

Max Activating Token Index: 719

Text #8

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.5177. Min Act: -0.0000

Data Index: 1591473 (Python Code)

Max Activating Token Index: 431

Text #9

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.5845. Min Act: -0.0000

Data Index: 1456703 (Python Code)

Max Activating Token Index: 898

Text #10

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.5542. Min Act: 0.0000

Data Index: 1558600 (Python Code)

Max Activating Token Index: 550

Text #11

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.4272. Min Act: -0.0000

Data Index: 1360773 (Python Code)

Max Activating Token Index: 542

Text #12

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.5465. Min Act: -0.0000

Data Index: 1549359 (Python Code)

Max Activating Token Index: 189

Text #13

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.3828. Min Act: -0.0000

Data Index: 1576943 (Python Code)

Max Activating Token Index: 321

Text #14

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.4619. Min Act: -0.0000

Data Index: 1591643 (Python Code)

Max Activating Token Index: 15

Text #15

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.4528. Min Act: -0.0000

Data Index: 1388338 (Python Code)

Max Activating Token Index: 920

Text #16

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.3063. Min Act: -0.0000

Data Index: 1564550 (Python Code)

Max Activating Token Index: 249

Text #17

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.4141. Min Act: -0.0000

Data Index: 1634873 (Python Code)

Max Activating Token Index: 775

Text #18

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.4119. Min Act: -0.0000

Data Index: 1380436 (Python Code)

Max Activating Token Index: 815

Text #19

Max Range: 2.8544. Min Range: -2.8544

Max Act: 2.3717. Min Act: -0.0000

Data Index: 1616512 (Python Code)

Max Activating Token Index: 92