Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3587 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 12.0589. Min Range: -12.0589

Max Act: 12.0027. Min Act: -0.0000

Data Index: 939884 (C4 (Web Text))

Max Activating Token Index: 866

Text #1

Max Range: 12.0589. Min Range: -12.0589

Max Act: 11.8044. Min Act: -0.0000

Data Index: 297434 (C4 (Web Text))

Max Activating Token Index: 314

Text #2

Max Range: 12.0589. Min Range: -12.0589

Max Act: 12.0589. Min Act: -0.0000

Data Index: 332440 (C4 (Web Text))

Max Activating Token Index: 270

Text #3

Max Range: 12.0589. Min Range: -12.0589

Max Act: 11.3504. Min Act: -0.0000

Data Index: 929146 (C4 (Web Text))

Max Activating Token Index: 315

Text #4

Max Range: 12.0589. Min Range: -12.0589

Max Act: 11.4932. Min Act: -0.0000

Data Index: 851520 (C4 (Web Text))

Max Activating Token Index: 920

Text #5

Max Range: 12.0589. Min Range: -12.0589

Max Act: 11.3364. Min Act: -0.0000

Data Index: 91114 (C4 (Web Text))

Max Activating Token Index: 638

Text #6

Max Range: 12.0589. Min Range: -12.0589

Max Act: 11.1034. Min Act: -0.0000

Data Index: 113829 (C4 (Web Text))

Max Activating Token Index: 634

Text #7

Max Range: 12.0589. Min Range: -12.0589

Max Act: 10.9162. Min Act: -0.0000

Data Index: 560291 (C4 (Web Text))

Max Activating Token Index: 462

Text #8

Max Range: 12.0589. Min Range: -12.0589

Max Act: 11.1682. Min Act: -0.0000

Data Index: 72168 (C4 (Web Text))

Max Activating Token Index: 339

Text #9

Max Range: 12.0589. Min Range: -12.0589

Max Act: 10.4536. Min Act: -0.0000

Data Index: 472613 (C4 (Web Text))

Max Activating Token Index: 247

Text #10

Max Range: 12.0589. Min Range: -12.0589

Max Act: 11.1159. Min Act: -0.0000

Data Index: 1244536 (C4 (Web Text))

Max Activating Token Index: 872

Text #11

Max Range: 12.0589. Min Range: -12.0589

Max Act: 11.0812. Min Act: -0.0000

Data Index: 194064 (C4 (Web Text))

Max Activating Token Index: 247

Text #12

Max Range: 12.0589. Min Range: -12.0589

Max Act: 10.4519. Min Act: -0.0000

Data Index: 1320977 (C4 (Web Text))

Max Activating Token Index: 997

Text #13

Max Range: 12.0589. Min Range: -12.0589

Max Act: 10.7589. Min Act: -0.0000

Data Index: 992973 (C4 (Web Text))

Max Activating Token Index: 919

Text #14

Max Range: 12.0589. Min Range: -12.0589

Max Act: 10.3583. Min Act: -0.0000

Data Index: 1016967 (C4 (Web Text))

Max Activating Token Index: 901

Text #15

Max Range: 12.0589. Min Range: -12.0589

Max Act: 10.0049. Min Act: -0.0000

Data Index: 260555 (C4 (Web Text))

Max Activating Token Index: 623

Text #16

Max Range: 12.0589. Min Range: -12.0589

Max Act: 10.1565. Min Act: -0.0000

Data Index: 466627 (C4 (Web Text))

Max Activating Token Index: 913

Text #17

Max Range: 12.0589. Min Range: -12.0589

Max Act: 10.7797. Min Act: -0.0000

Data Index: 1200999 (C4 (Web Text))

Max Activating Token Index: 1001

Text #18

Max Range: 12.0589. Min Range: -12.0589

Max Act: 10.4291. Min Act: -0.0000

Data Index: 569702 (C4 (Web Text))

Max Activating Token Index: 768

Text #19

Max Range: 12.0589. Min Range: -12.0589

Max Act: 10.4311. Min Act: -0.0000

Data Index: 636970 (C4 (Web Text))

Max Activating Token Index: 481