Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1038 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.7167. Min Act: -0.0000

Data Index: 1111599 (C4 (Web Text))

Max Activating Token Index: 751

Text #1

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.7072. Min Act: -0.0000

Data Index: 949186 (C4 (Web Text))

Max Activating Token Index: 731

Text #2

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.6720. Min Act: -0.0000

Data Index: 904921 (C4 (Web Text))

Max Activating Token Index: 389

Text #3

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.6347. Min Act: 0.0000

Data Index: 729795 (C4 (Web Text))

Max Activating Token Index: 155

Text #4

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5818. Min Act: -0.0000

Data Index: 839667 (C4 (Web Text))

Max Activating Token Index: 195

Text #5

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5763. Min Act: 0.0000

Data Index: 27207 (C4 (Web Text))

Max Activating Token Index: 399

Text #6

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5854. Min Act: 0.0001

Data Index: 382658 (C4 (Web Text))

Max Activating Token Index: 95

Text #7

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5629. Min Act: -0.0000

Data Index: 391851 (C4 (Web Text))

Max Activating Token Index: 1000

Text #8

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5429. Min Act: 0.0001

Data Index: 908455 (C4 (Web Text))

Max Activating Token Index: 918

Text #9

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5137. Min Act: -0.0000

Data Index: 888830 (C4 (Web Text))

Max Activating Token Index: 246

Text #10

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5473. Min Act: -0.0000

Data Index: 1163958 (C4 (Web Text))

Max Activating Token Index: 210

Text #11

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5385. Min Act: -0.0000

Data Index: 122942 (C4 (Web Text))

Max Activating Token Index: 21

Text #12

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5178. Min Act: 0.0000

Data Index: 587445 (C4 (Web Text))

Max Activating Token Index: 460

Text #13

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5163. Min Act: -0.0000

Data Index: 221364 (C4 (Web Text))

Max Activating Token Index: 872

Text #14

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5407. Min Act: -0.0000

Data Index: 1025479 (C4 (Web Text))

Max Activating Token Index: 333

Text #15

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5211. Min Act: -0.0000

Data Index: 711991 (C4 (Web Text))

Max Activating Token Index: 378

Text #16

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.4878. Min Act: -0.0000

Data Index: 1204514 (C4 (Web Text))

Max Activating Token Index: 60

Text #17

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.5010. Min Act: -0.0000

Data Index: 858590 (C4 (Web Text))

Max Activating Token Index: 879

Text #18

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.4956. Min Act: -0.0000

Data Index: 506492 (C4 (Web Text))

Max Activating Token Index: 921

Text #19

Max Range: 0.7167. Min Range: -0.7167

Max Act: 0.4914. Min Act: 0.0000

Data Index: 707107 (C4 (Web Text))

Max Activating Token Index: 605