Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3289 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1802. Min Act: -0.0001

Data Index: 1007961 (C4 (Web Text))

Max Activating Token Index: 243

Text #1

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1463. Min Act: -0.0000

Data Index: 634131 (C4 (Web Text))

Max Activating Token Index: 354

Text #2

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1516. Min Act: -0.0000

Data Index: 445346 (C4 (Web Text))

Max Activating Token Index: 358

Text #3

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1637. Min Act: -0.0000

Data Index: 633364 (C4 (Web Text))

Max Activating Token Index: 401

Text #4

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1499. Min Act: -0.0000

Data Index: 180085 (C4 (Web Text))

Max Activating Token Index: 593

Text #5

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1661. Min Act: -0.0001

Data Index: 369559 (C4 (Web Text))

Max Activating Token Index: 204

Text #6

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1410. Min Act: -0.0000

Data Index: 589472 (C4 (Web Text))

Max Activating Token Index: 448

Text #7

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1556. Min Act: -0.0000

Data Index: 806163 (C4 (Web Text))

Max Activating Token Index: 352

Text #8

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1857. Min Act: -0.0000

Data Index: 291030 (C4 (Web Text))

Max Activating Token Index: 1018

Text #9

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1536. Min Act: -0.0000

Data Index: 844932 (C4 (Web Text))

Max Activating Token Index: 784

Text #10

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1892. Min Act: -0.0000

Data Index: 515406 (C4 (Web Text))

Max Activating Token Index: 983

Text #11

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1673. Min Act: -0.0000

Data Index: 887759 (C4 (Web Text))

Max Activating Token Index: 585

Text #12

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1414. Min Act: -0.0000

Data Index: 406627 (C4 (Web Text))

Max Activating Token Index: 840

Text #13

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1359. Min Act: -0.0000

Data Index: 343820 (C4 (Web Text))

Max Activating Token Index: 947

Text #14

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1708. Min Act: -0.0000

Data Index: 970933 (C4 (Web Text))

Max Activating Token Index: 755

Text #15

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1210. Min Act: -0.0000

Data Index: 756076 (C4 (Web Text))

Max Activating Token Index: 695

Text #16

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1475. Min Act: -0.0000

Data Index: 749236 (C4 (Web Text))

Max Activating Token Index: 914

Text #17

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1614. Min Act: -0.0000

Data Index: 960636 (C4 (Web Text))

Max Activating Token Index: 598

Text #18

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1519. Min Act: -0.0000

Data Index: 396602 (C4 (Web Text))

Max Activating Token Index: 400

Text #19

Max Range: 0.1892. Min Range: -0.1892

Max Act: 0.1457. Min Act: -0.0000

Data Index: 208031 (C4 (Web Text))

Max Activating Token Index: 1011