Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3869 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.4013. Min Act: -0.0000

Data Index: 1136239 (C4 (Web Text))

Max Activating Token Index: 671

Text #1

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.4164. Min Act: -0.0000

Data Index: 277606 (C4 (Web Text))

Max Activating Token Index: 504

Text #2

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3923. Min Act: -0.0000

Data Index: 191103 (C4 (Web Text))

Max Activating Token Index: 60

Text #3

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3849. Min Act: -0.0000

Data Index: 618011 (C4 (Web Text))

Max Activating Token Index: 972

Text #4

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3938. Min Act: -0.0000

Data Index: 111188 (C4 (Web Text))

Max Activating Token Index: 474

Text #5

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3838. Min Act: -0.0000

Data Index: 371082 (C4 (Web Text))

Max Activating Token Index: 497

Text #6

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3818. Min Act: -0.0000

Data Index: 216037 (C4 (Web Text))

Max Activating Token Index: 410

Text #7

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3593. Min Act: -0.0000

Data Index: 950922 (C4 (Web Text))

Max Activating Token Index: 476

Text #8

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3524. Min Act: -0.0000

Data Index: 204628 (C4 (Web Text))

Max Activating Token Index: 416

Text #9

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3527. Min Act: -0.0000

Data Index: 711441 (C4 (Web Text))

Max Activating Token Index: 123

Text #10

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3720. Min Act: -0.0000

Data Index: 615737 (C4 (Web Text))

Max Activating Token Index: 497

Text #11

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3729. Min Act: -0.0000

Data Index: 1170987 (C4 (Web Text))

Max Activating Token Index: 339

Text #12

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3496. Min Act: 0.0000

Data Index: 330262 (C4 (Web Text))

Max Activating Token Index: 255

Text #13

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3335. Min Act: -0.0000

Data Index: 734012 (C4 (Web Text))

Max Activating Token Index: 656

Text #14

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3512. Min Act: -0.0000

Data Index: 653709 (C4 (Web Text))

Max Activating Token Index: 765

Text #15

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.2910. Min Act: -0.0000

Data Index: 313301 (C4 (Web Text))

Max Activating Token Index: 804

Text #16

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3491. Min Act: -0.0000

Data Index: 1354217 (C4 (Web Text))

Max Activating Token Index: 1022

Text #17

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3271. Min Act: 0.0000

Data Index: 1196912 (C4 (Web Text))

Max Activating Token Index: 407

Text #18

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3238. Min Act: 0.0000

Data Index: 1168032 (C4 (Web Text))

Max Activating Token Index: 241

Text #19

Max Range: 0.4164. Min Range: -0.4164

Max Act: 0.3332. Min Act: -0.0000

Data Index: 618309 (C4 (Web Text))

Max Activating Token Index: 363