Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 68 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.3304. Min Act: -0.0000

Data Index: 694510 (C4 (Web Text))

Max Activating Token Index: 782

Text #1

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.3054. Min Act: -0.0000

Data Index: 1316021 (C4 (Web Text))

Max Activating Token Index: 805

Text #2

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.3005. Min Act: -0.0000

Data Index: 1243261 (C4 (Web Text))

Max Activating Token Index: 414

Text #3

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.3151. Min Act: -0.0000

Data Index: 3913 (C4 (Web Text))

Max Activating Token Index: 546

Text #4

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2825. Min Act: -0.0000

Data Index: 1320530 (C4 (Web Text))

Max Activating Token Index: 539

Text #5

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2938. Min Act: -0.0000

Data Index: 837208 (C4 (Web Text))

Max Activating Token Index: 919

Text #6

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2891. Min Act: -0.0000

Data Index: 877316 (C4 (Web Text))

Max Activating Token Index: 314

Text #7

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2707. Min Act: -0.0000

Data Index: 1307182 (C4 (Web Text))

Max Activating Token Index: 928

Text #8

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2359. Min Act: -0.0000

Data Index: 276426 (C4 (Web Text))

Max Activating Token Index: 199

Text #9

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2604. Min Act: -0.0000

Data Index: 401187 (C4 (Web Text))

Max Activating Token Index: 98

Text #10

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2523. Min Act: -0.0000

Data Index: 948199 (C4 (Web Text))

Max Activating Token Index: 449

Text #11

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2742. Min Act: -0.0000

Data Index: 1120786 (C4 (Web Text))

Max Activating Token Index: 588

Text #12

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2468. Min Act: -0.0000

Data Index: 294525 (C4 (Web Text))

Max Activating Token Index: 664

Text #13

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2614. Min Act: -0.0000

Data Index: 999576 (C4 (Web Text))

Max Activating Token Index: 99

Text #14

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2451. Min Act: -0.0000

Data Index: 795596 (C4 (Web Text))

Max Activating Token Index: 155

Text #15

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2324. Min Act: -0.0001

Data Index: 157391 (C4 (Web Text))

Max Activating Token Index: 829

Text #16

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2481. Min Act: -0.0000

Data Index: 680299 (C4 (Web Text))

Max Activating Token Index: 762

Text #17

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2535. Min Act: -0.0000

Data Index: 1093951 (C4 (Web Text))

Max Activating Token Index: 666

Text #18

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2311. Min Act: -0.0000

Data Index: 295623 (C4 (Web Text))

Max Activating Token Index: 281

Text #19

Max Range: 0.3304. Min Range: -0.3304

Max Act: 0.2378. Min Act: -0.0000

Data Index: 1037207 (C4 (Web Text))

Max Activating Token Index: 136