Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 619 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.3779. Min Range: -2.3779

Max Act: 2.3779. Min Act: -0.0000

Data Index: 536298 (C4 (Web Text))

Max Activating Token Index: 240

Text #1

Max Range: 2.3779. Min Range: -2.3779

Max Act: 2.1016. Min Act: -0.0000

Data Index: 142578 (C4 (Web Text))

Max Activating Token Index: 642

Text #2

Max Range: 2.3779. Min Range: -2.3779

Max Act: 2.1948. Min Act: -0.0000

Data Index: 992209 (C4 (Web Text))

Max Activating Token Index: 742

Text #3

Max Range: 2.3779. Min Range: -2.3779

Max Act: 2.1169. Min Act: -0.0000

Data Index: 708777 (C4 (Web Text))

Max Activating Token Index: 428

Text #4

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.8855. Min Act: -0.0000

Data Index: 266837 (C4 (Web Text))

Max Activating Token Index: 559

Text #5

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.9305. Min Act: -0.0000

Data Index: 929854 (C4 (Web Text))

Max Activating Token Index: 406

Text #6

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.8521. Min Act: -0.0000

Data Index: 448474 (C4 (Web Text))

Max Activating Token Index: 367

Text #7

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.6958. Min Act: -0.0000

Data Index: 1234269 (C4 (Web Text))

Max Activating Token Index: 375

Text #8

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.7286. Min Act: -0.0000

Data Index: 1318411 (C4 (Web Text))

Max Activating Token Index: 534

Text #9

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.7216. Min Act: -0.0000

Data Index: 230258 (C4 (Web Text))

Max Activating Token Index: 464

Text #10

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.6965. Min Act: -0.0000

Data Index: 1230705 (C4 (Web Text))

Max Activating Token Index: 273

Text #11

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.6911. Min Act: -0.0000

Data Index: 417018 (C4 (Web Text))

Max Activating Token Index: 670

Text #12

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.6992. Min Act: -0.0000

Data Index: 770246 (C4 (Web Text))

Max Activating Token Index: 832

Text #13

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.6010. Min Act: -0.0000

Data Index: 523901 (C4 (Web Text))

Max Activating Token Index: 481

Text #14

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.6431. Min Act: -0.0000

Data Index: 778115 (C4 (Web Text))

Max Activating Token Index: 481

Text #15

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.6216. Min Act: -0.0000

Data Index: 1044133 (C4 (Web Text))

Max Activating Token Index: 462

Text #16

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.5612. Min Act: -0.0000

Data Index: 993053 (C4 (Web Text))

Max Activating Token Index: 365

Text #17

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.5935. Min Act: -0.0000

Data Index: 1279150 (C4 (Web Text))

Max Activating Token Index: 449

Text #18

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.5476. Min Act: -0.0000

Data Index: 237104 (C4 (Web Text))

Max Activating Token Index: 331

Text #19

Max Range: 2.3779. Min Range: -2.3779

Max Act: 1.5553. Min Act: -0.0000

Data Index: 106941 (C4 (Web Text))

Max Activating Token Index: 594